MinIO Setup¶
This section provides instructions for configuring Driverless AI to work with MinIO. Note that unlike S3, authentication must also be configured when the MinIO data connector is specified.
Note: Depending on your Docker install version, use either the docker run --runtime=nvidia (>= Docker 19.03) or nvidia-docker (< Docker 19.03) command when starting the Driverless AI Docker image. Use docker version to check which version of Docker you are using.
Description of Configuration Attributes¶
minio_endpoint_url: The endpoint URL that will be used to access MinIO.minio_access_key_id: The MinIO access key.minio_secret_access_key: The MinIO secret access key.minio_skip_cert_verification: If this is set to true, then MinIO connector will skip certificate verification. This is set to false by default.enabled_file_systems: The file systems you want to enable. This must be configured in order for data connectors to function properly.
Enable MinIO with Authentication¶
This example enables the MinIO data connector with authentication by passing an endpoint URL, access key ID, and an access key. It also configures Docker DNS by passing the name and IP of the name node. This lets you reference data stored in MinIO directly using the endpoint URL, for example: http://<endpoint_url>/<bucket>/datasets/iris.csv.
nvidia-docker run \
--shm-size=2g --cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit nproc=16384:16384 \
--add-host name.node:172.16.2.186 \
-e DRIVERLESS_AI_ENABLED_FILE_SYSTEMS="file,minio" \
-e DRIVERLESS_AI_MINIO_ENDPOINT_URL="<endpoint_url>"
-e DRIVERLESS_AI_MINIO_ACCESS_KEY_ID="<access_key_id>" \
-e DRIVERLESS_AI_MINIO_SECRET_ACCESS_KEY="<access_key>" \
-e DRIVERLESS_AI_MINIO_SKIP_CERT_VERIFICATION="false" \
-p 12345:12345 \
--init -it --rm \
-v /tmp/dtmp/:/tmp \
-v /tmp/dlog/:/log \
-v /tmp/dlicense/:/license \
-v /tmp/ddata/:/data \
-u $(id -u):$(id -g) \
h2oai/dai-ubi8-x86_64:1.11.1-cuda11.8.0.xx
This example shows how to configure MinIO options in the config.toml file, and then specify that file when starting Driverless AI in Docker.
Configure the Driverless AI config.toml file. Set the following configuration options.
enabled_file_systems = "file, upload, minio"
minio_endpoint_url = "<endpoint_url>"
minio_access_key_id = "<access_key_id>"
minio_secret_access_key = "<access_key>"
minio_skip_cert_verification = "false"
Mount the config.toml file into the Docker container.
nvidia-docker run \ --pid=host \ --init \ --rm \ --shm-size=2g --cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit nproc=16384:16384 \ --add-host name.node:172.16.2.186 \ -e DRIVERLESS_AI_CONFIG_FILE=/path/in/docker/config.toml \ -p 12345:12345 \ -v /local/path/to/config.toml:/path/in/docker/config.toml \ -v /etc/passwd:/etc/passwd:ro \ -v /etc/group:/etc/group:ro \ -v /tmp/dtmp/:/tmp \ -v /tmp/dlog/:/log \ -v /tmp/dlicense/:/license \ -v /tmp/ddata/:/data \ -u $(id -u):$(id -g) \ h2oai/dai-ubi8-x86_64:1.11.1-cuda11.8.0.xx
This example enables the MinIO data connector with authentication by passing an endpoint URL, access key ID, and an access key. It also configures Docker DNS by passing the name and IP of the MinIO endpoint. This allows users to reference data stored in MinIO directly using the endpoint URL, for example: http://<endpoint_url>/<bucket>/datasets/iris.csv.
Export the Driverless AI config.toml file or add it to ~/.bashrc. For example:
# DEB and RPM export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml" # TAR SH export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"
Specify the following configuration options in the config.toml file.
# File System Support # upload : standard upload feature # file : local file system/server file system # hdfs : Hadoop file system, remember to configure the HDFS config folder path and keytab below # dtap : Blue Data Tap file system, remember to configure the DTap section below # s3 : Amazon S3, optionally configure secret and access key below # gcs : Google Cloud Storage, remember to configure gcs_path_to_service_account_json below # gbq : Google Big Query, remember to configure gcs_path_to_service_account_json below # minio : MinIO Cloud Storage, remember to configure secret and access key below # snow : Snowflake Data Warehouse, remember to configure Snowflake credentials below (account name, username, password) # kdb : KDB+ Time Series Database, remember to configure KDB credentials below (hostname and port, optionally: username, password, classpath, and jvm_args) # azrbs : Azure Blob Storage, remember to configure Azure credentials below (account name, account key) # jdbc: JDBC Connector, remember to configure JDBC below. (jdbc_app_configs) # hive: Hive Connector, remember to configure Hive below. (hive_app_configs) # recipe_url: load custom recipe from URL # recipe_file: load custom recipe from local file system enabled_file_systems = "file, minio" # MinIO Connector credentials minio_endpoint_url = "<endpoint_url>" minio_access_key_id = "<access_key_id>" minio_secret_access_key = "<access_key>" minio_skip_cert_verification = "false"
Save the changes when you are done, then stop/restart Driverless AI.