S3 Setup

Driverless AI lets you explore S3 data sources from within the Driverless AI application. This section provides instructions for configuring Driverless AI to work with S3.

Note: Depending on your Docker install version, use either the docker run --runtime=nvidia (>= Docker 19.03) or nvidia-docker (< Docker 19.03) command when starting the Driverless AI Docker image. Use docker version to check which version of Docker you are using.

Description of Configuration Attributes

• aws_access_key_id: The S3 access key ID

• aws_secret_access_key: The S3 access key

• aws_role_arn: The Amazon Resource Name

• aws_default_region: The region to use when the aws_s3_endpoint_url option is not set. This is ignored when aws_s3_endpoint_url is set.

• aws_s3_endpoint_url: The endpoint URL that will be used to access S3.

• aws_use_ec2_role_credentials: If set to true, the S3 Connector will try to to obtain credentials associated with the role attached to the EC2 instance.

• s3_init_path: The starting S3 path that will be displayed in UI S3 browser.

• enabled_file_systems: The file systems you want to enable. This must be configured in order for data connectors to function properly.

Example 1: Enable S3 with No Authentication

Docker Image InstallsDocker Image with the config.tomlNative Installs

This example enables the S3 data connector and disables authentication. It does not pass any S3 access key or secret; however it configures Docker DNS by passing the name and IP of the S3 name node. This allows users to reference data stored in S3 directly using the name node address, for example: s3://name.node/datasets/iris.csv.

 nvidia-docker run \
            --shm-size=2g --cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit nproc=16384:16384 \
            --add-host name.node:172.16.2.186 \
            -e DRIVERLESS_AI_ENABLED_FILE_SYSTEMS="file,s3" \
            -p 12345:12345 \
            --init -it --rm \
            -v /tmp/dtmp/:/tmp \
            -v /tmp/dlog/:/log \
            -v /tmp/dlicense/:/license \
            -v /tmp/ddata/:/data \
            -u $(id -u):$(id -g) \
            h2oai/dai-ubi8-x86_64:1.11.1-cuda11.8.0.xx

Example 2: Enable S3 with Authentication

Docker Image InstallsDocker Image with the config.tomlNative Installs

This example enables the S3 data connector with authentication by passing an S3 access key ID and an access key. It also configures Docker DNS by passing the name and IP of the S3 name node. This allows users to reference data stored in S3 directly using the name node address, for example: s3://name.node/datasets/iris.csv.

      nvidia-docker run \
                     --shm-size=2g --cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit nproc=16384:16384 \
                     --add-host name.node:172.16.2.186 \
                     -e DRIVERLESS_AI_ENABLED_FILE_SYSTEMS="file,s3" \
                     -e DRIVERLESS_AI_AWS_ACCESS_KEY_ID="<access_key_id>" \
                     -e DRIVERLESS_AI_AWS_SECRET_ACCESS_KEY="<access_key>" \
                     -p 12345:12345 \
                     --init -it --rm \
                     -v /tmp/dtmp/:/tmp \
                     -v /tmp/dlog/:/log \
                     -v /tmp/dlicense/:/license \
                     -v /tmp/ddata/:/data \
                     -u $(id -u):$(id -g) \
                     h2oai/dai-ubi8-x86_64:1.11.1-cuda11.8.0.xx