Configuration Security

Driverless AI provides the option to store sensitive or secure configuration information in an encrypted keystore as an alternative to keeping security settings as clear text in the config.toml file.

Updates to config override chain

The Configuration Override Chain has been updated to load the settings from the encrypted keystore after the settings are read from the plain text config.toml file. The Environment Variable can still override the values from the keystore:

1. h2oai/config/config.toml
[Internal, not visible to users]

2. config.toml
[Place file in a folder/mount file in docker container and provide path
in "DRIVERLESS_AI_CONFIG_FILE" environment variable]

3. Keystore file
[Set keystore_file parameter in config.toml or environment variable
"DRIVERLESS_AI_KEYSTORE_FILE" to point to a valid DAI keystore file
generated using the h2oai.keystore tool. If env variable is set, the value
in the config.toml for keystore_file path is overridden]

4. Environment variable
[Configuration variables can also be provided as environment
variables. They must have the prefix "DRIVERLESS_AI_" followed
by the variable name in caps. For example, "authentication_method"
can be provided as "DRIVERLESS_AI_AUTHENTICATION_METHOD"]

Keystore setup workflow

Creating the keystore

Although the keystore file can contain any configuration parameter supported by the config.toml, it is recommended to store only config parameters that contain secure/sensitive information in the keystore file and use the regular config.toml file for other config parameters.

Step 1: Create a cleartext config subset

To start, create a file config.clear that follows the TOML syntax of a regular config.toml file and contains the config parameters that you want to store securely. For example:

vagrant@ubuntu-bionic:~$ cat /home/vagrant/config.clear
# ldap connection details
ldap_bind_password = "somepassword"
# Snowflake Connector credentials
snowflake_url = "https://sampleurl"
snowflake_user = "sampleuser"
snowflake_password = "samplepass"
snowflake_account = "sampleaccount"
vagrant@ubuntu-bionic:~$

Step 2: Using the h2oai.keystore tool to create keystore

The keystore should be placed so that it is accessible by root or the user id with which the Driverless AI process is running. We recommend storing the keystore as /etc/dai/config.keystore along with all other Driverless configuration files.

To create a keystore from the config.clear file, use the h2oai.keystore tool:

  • The keystore tool needs to be run as root and within the context of Driverless AI Python environment provided by the dai-env.sh script.

  • The add-keys command accepts the path to keystore as the first argument and the clear text config.toml subset as the second.

  • If the keystore does not exist, it is created.

  • All keys in the config.clear are either Inserted or Updated in the keystore. If a key already exists in the key store, it is updated. If the keystore contains any keys that are not in config.clear, they are not altered.

  • Once the keystore file is created, it is recommended to ensure the following:

    • Ownership is with root user with read and write permissions.

    • Change group ownership to the Driverless group (or the appropriate ID that matches the group ID with which the Driverless processes run in your system) with read only permissions. No other user or group should have read access to this file.

  • The config.keystore file is created along with the ownership permissions.

(user1) $ sudo /bin/bash    # this will get a shell as root. If root access shell is available; this step can be skipped
(root) # /opt/h2oai/dai/dai-env.sh python -m h2oai.keystore add-keys /etc/dai/config.keystore /home/vagrant/config.clear
....some output here
======================================================================
Key: ldap_bind_password; Action: Inserted
Key: snowflake_url; Action: Inserted
Key: snowflake_user; Action: Inserted
Key: snowflake_password; Action: Inserted
Key: snowflake_account; Action: Inserted

(root) # ls -l /etc/dai
total 240
-rw-rw-r-- 1 root root    353 Jul 14 03:28 EnvironmentFile.conf
-rw-r--r-- 1 root root    210 Jul 20 06:57 Group.conf
-rw-r--r-- 1 root root    209 Jul 20 06:57 User.conf
-rw-r----- 1 root dai     236 Jul 20 07:09 config.keystore
-rw-r--r-- 1 root root 157135 Jul 20 07:17 config.toml
-rw-rw-r-- 1 root root    347 Jul 14 03:28 jaas.conf
-rw-r--r-- 1 root root  62206 Jul 20 06:57 redis.conf

(root) # chown root:dai /etc/dai/config.keystore
(root) # chmod 640 /etc/dai/config.keystore

Step 3: Using h2oai.keystore tool to manage keystore

The h2oai.keystore tool provides three commands for keystore management:

  • add-keys: Adds or updates the Driverless AI secrets keystore with config.

  • list-keys: Lists the keys stored in the Driverless AI keystore. Their values are never displayed.

  • delete-keys Removes the specified key from the keystore.

(root) # /opt/h2oai/dai/dai-env.sh python -m h2oai.keystore --help

======================================================================
Usage: python -m h2oai.keystore [OPTIONS] COMMAND [ARGS]...

Options:
 --help  Show this message and exit.

Commands:
 add-keys     Adds/Update DAI secrets keystore (KEYSTORE_PATH) with config...
 delete-keys  Deleted keys provided as -k key1 -k key2 from the keystore...
 list-keys    Lists keys stored in Driverless AI KEYSTORE.
root@ubuntu-bionic:/etc/dai#


(root) # /opt/h2oai/dai/dai-env.sh python -m h2oai.keystore list-keys /etc/dai/config.keystore
======================================================================
ldap_bind_password = ******
snowflake_url = ******
snowflake_user = ******
snowflake_password = ******
snowflake_account = ******
root@ubuntu-bionic:/etc/dai#

-------- Deleting keys ------------

(root) # /opt/h2oai/dai/dai-env.sh python -m h2oai.keystore delete-keys /etc/dai/config.keystore  -k snowflake_url -k snowflake_account
======================================================================
snowflake_url = Deleted
snowflake_account = Deleted

(root) # /opt/h2oai/dai/dai-env.sh python -m h2oai.keystore list-keys /etc/dai/config.keystore
======================================================================
ldap_bind_password = ******
snowflake_user = ******
snowflake_password = ******
root@ubuntu-bionic:/etc/dai#