Minio Setup

This section provides instructions for configuring Driverless AI to work with Minio. Note that unlike S3, authentication must also be configured when the Minio data connector is specified.

Description of Configuration Attributes

  • minio_endpoint_url: The endpoint URL that will be used to access Minio.

  • minio_access_key_id: The Minio access key.

  • minio_secret_access_key: The Minio secret access key.

  • enabled_file_systems: The file systems you want to enable. This must be configured in order for data connectors to function properly.

Minio with Authentication

This example enables the Minio data connector with authentication by passing an endpoint URL, access key ID, and an access key. It also configures Docker DNS by passing the name and IP of the Minio endpoint. This allows users to reference data stored in Minio directly using the endpoint URL, for example: http://<endpoint_url>/<bucket>/datasets/iris.csv.

  1. Export the Driverless AI config.toml file or add it to ~/.bashrc. For example:

# DEB and RPM
export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"

# TAR SH
export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"

  1. Specify the following configuration options in the config.toml file.

# File System Support
# upload : standard upload feature
# file : local file system/server file system
# hdfs : Hadoop file system, remember to configure the HDFS config folder path and keytab below
# dtap : Blue Data Tap file system, remember to configure the DTap section below
# s3 : Amazon S3, optionally configure secret and access key below
# gcs: Google Cloud Storage, remember to configure gcs_path_to_service_account_json below
# gbq: Google Big Query, remember to configure gcs_path_to_service_account_json below
# minio: Minio Cloud Storage, remember to configure secret and access key below
# snow: Snowflake Data Warehouse, remember to configure Snowflake credentials below (account name, username, password)
# kdb: KDB+ Time Series Database, remember to configure KDB credentials below (hostname and port, optionally: username, password, classpath, and jvm_args)
# azrbs: Azure Blob Storage, remember to configure Azure credentials below (account name, account key)
# jdbc: JDBC Connector, remember to configure JDBC below. (jdbc_app_configs)
# hive: Hive Connector, remember to configure Hive below. (hive_app_configs)
# recipe_url: load custom recipe from URL
# recipe_file: load custom recipe from local file system
enabled_file_systems = "file, minio"

# Minio Connector credentials
minio_endpoint_url = "<endpoint_url>"
minio_access_key_id = "<access_key_id>"
minio_secret_access_key = "<access_key>"

  1. Save the changes when you are done, then stop/restart Driverless AI.