MinIO Setup

This section provides instructions for configuring Driverless AI to work with MinIO. Note that unlike S3, authentication must also be configured when the MinIO data connector is specified.

Note: Depending on your Docker install version, use either the docker run --runtime=nvidia (>= Docker 19.03) or nvidia-docker (< Docker 19.03) command when starting the Driverless AI Docker image. Use docker version to check which version of Docker you are using.

Description of Configuration Attributes

  • minio_endpoint_url: The endpoint URL that will be used to access MinIO.

  • minio_access_key_id: The MinIO access key.

  • minio_secret_access_key: The MinIO secret access key.

  • minio_skip_cert_verification: If this is set to true, then MinIO connector will skip certificate verification. This is set to false by default.

  • enabled_file_systems: The file systems you want to enable. This must be configured in order for data connectors to function properly.

Enable MinIO with Authentication

This example enables the MinIO data connector with authentication by passing an endpoint URL, access key ID, and an access key. It also configures Docker DNS by passing the name and IP of the name node. This lets you reference data stored in MinIO directly using the endpoint URL, for example: http://<endpoint_url>/<bucket>/datasets/iris.csv.

nvidia-docker run \
 --shm-size=2g --cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit nproc=16384:16384 \
 --add-host name.node:172.16.2.186 \
 -e DRIVERLESS_AI_ENABLED_FILE_SYSTEMS="file,minio" \
 -e DRIVERLESS_AI_MINIO_ENDPOINT_URL="<endpoint_url>"
 -e DRIVERLESS_AI_MINIO_ACCESS_KEY_ID="<access_key_id>" \
 -e DRIVERLESS_AI_MINIO_SECRET_ACCESS_KEY="<access_key>" \
 -e DRIVERLESS_AI_MINIO_SKIP_CERT_VERIFICATION="false" \
 -p 12345:12345 \
 --init -it --rm \
 -v /tmp/dtmp/:/tmp \
 -v /tmp/dlog/:/log \
 -v /tmp/dlicense/:/license \
 -v /tmp/ddata/:/data \
 -u $(id -u):$(id -g) \
 h2oai/dai-ubi8-x86_64:1.11.1-cuda11.8.0.xx