S3 Setup

Driverless AI allows you to explore S3 data sources from within the Driverless AI application. This section provides instructions for configuring Driverless AI to work with S3.

Description of Configuration Attributes

  • aws_access_key_id: The S3 access key ID
  • aws_secret_access_key: The S3 access key

S3 with No Authentication

This example enables the S3 data connector and disables authentication. It does not pass any S3 access key or secret; however it configures Docker DNS by passing the name and IP of the S3 name node. This allows users to reference data stored in S3 directly using the name node address, for example: s3://name.node/datasets/iris.csv. Replace TAG below with the image tag.

nvidia-docker run \
            --shm-size=256m \
            --add-host name.node:172.16.2.186 \
            -e DRIVERLESS_AI_ENABLED_FILE_SYSTEMS="file,s3" \
            -p 12345:12345 \
            --init -it --rm \
            -v /tmp/dtmp/:/tmp \
            -v /tmp/dlog/:/log \
            -v /tmp/dlicense/:/license \
            -v /tmp/ddata/:/data \
            -u $(id -u):$(id -g) \
            h2oai/dai-centos7-x86_64:TAG

S3 with Authentication

This example enables the S3 data connector with authentication by passing an S3 access key ID and an access key. It also configures Docker DNS by passing the name and IP of the S3 name node. This allows users to reference data stored in S3 directly using the name node address, for example: s3://name.node/datasets/iris.csv. Replace TAG below with the image tag.

nvidia-docker run \
        --shm-size=256m \
        --add-host name.node:172.16.2.186 \
        -e DRIVERLESS_AI_ENABLED_FILE_SYSTEMS="file,s3" \
        -e DRIVERLESS_AI_AWS_AUTH="True"
        -e DRIVERLESS_AI_AWS_ACCESS_KEY_ID="<access_key_id>" \
        -e DRIVERLESS_AI_AWS_SECRET_ACCESS_KEY="<access_key>" \
        -p 12345:12345 \
        --init -it --rm \
        -v /tmp/dtmp/:/tmp \
        -v /tmp/dlog/:/log \
        -v /tmp/dlicense/:/license \
        -v /tmp/ddata/:/data \
        -u $(id -u):$(id -g) \
        h2oai/dai-centos7-x86_64:TAG