Azure Blob Store Setup

Driverless AI allows you to explore Azure Blob Store data sources from within the Driverless AI application. This section describes how to enable the Azure Blob Store data connector in native install environments.

Supported Data Sources Using the Azure Blob Store Connector

The following data sources can be used with the Azure Blob Store connector.

The following data sources can be used with the Azure Blob Store connector when also using the HDFS connector.

Description of Configuration Attributes

The following configuration attributes are specific to enabling Azure Blob Storage.

  • azure_blob_account_name: The Microsoft Azure Storage account name. This should be the dns prefix created when the account was created (for example, “mystorage”).

  • azure_blob_account_key: Specify the account key that maps to your account name.

  • azure_connection_string: Optionally specify a new connection string. With this option, you can include an override for a host, port, and/or account name. For example,

    azure_connection_string = "DefaultEndpointsProtocol=http;AccountName=<account_name>;AccountKey=<account_key>;BlobEndpoint=http://<host>:<port>/<account_name>;"

  • azure_blob_init_path: Specifies the starting Azure Blob store path displayed in the UI of the Azure Blob store browser.

  • enabled_file_systems: The file systems you want to enable. This must be configured in order for data connectors to function properly.

The following additional configuration attributes can be used for enabling an HDFS Connector to connect to Azure Data Lake Gen 1 (and optionally with Azure Data Lake Gen 2).

  • hdfs_config_path: The location the HDFS config folder path. This folder can contain multiple config files.

  • hdfs_app_classpath: The HDFS classpath.

  • hdfs_app_supported_schemes: Supported schemas list is used as an initial check to ensure valid input to connector.

Example 1: Enabling the Azure Blob Store Data Connector

This example enables the Azure Blob Store data connector. This allows users to reference data stored on your Azure storage account using the account name, for example: https://mystorage.blob.core.windows.net.

  1. Export the Driverless AI config.toml file or add it to ~/.bashrc. For example:

# DEB and RPM
export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"

# TAR SH
export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"

  1. Specify the following configuration options in the config.toml file.

# File System Support
# upload : standard upload feature
# file : local file system/server file system
# hdfs : Hadoop file system, remember to configure the HDFS config folder path and keytab below
# dtap : Blue Data Tap file system, remember to configure the DTap section below
# s3 : Amazon S3, optionally configure secret and access key below
# gcs: Google Cloud Storage, remember to configure gcs_path_to_service_account_json below
# gbq: Google Big Query, remember to configure gcs_path_to_service_account_json below
# minio: Minio Cloud Storage, remember to configure secret and access key below
# snow: Snowflake Data Warehouse, remember to configure Snowflake credentials below (account name, username, password)
# kdb: KDB+ Time Series Database, remember to configure KDB credentials below (hostname and port, optionally: username, password, classpath, and jvm_args)
# azrbs: Azure Blob Storage, remember to configure Azure credentials below (account name, account key)
# jdbc: JDBC Connector, remember to configure JDBC below. (jdbc_app_configs)
# hive: Hive Connector, remember to configure Hive below. (hive_app_configs)
# recipe_url: load custom recipe from URL
# recipe_file: load custom recipe from local file system
enabled_file_systems = "file, azrbs"

# Azure Blob Store Connector credentials
azure_blob_account_name = "mystorage"
azure_blob_account_key = "<account_key>"

  1. Save the changes when you are done, then stop/restart Driverless AI.

Example 2: Mount Azure File Shares to the Local File System

Supported Data Sources Using the Local File System

  • Azure Files (File Storage)

Mounting Azure File Shares

Azure file shares can be mounted into the Local File system of Driverless AI. To mount the Azure file share, follow the steps listed on https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-linux.

Example 3: Enable HDFS Connector to Connect to Azure Data Lake Gen 1

This example enables the HDFS Connector to connect to Azure Data Lake Gen1. This allows users to reference data stored on your Azure Data Lake using the adl uri, for example: adl://myadl.azuredatalakestore.net.

This example enables the HDFS Connector to connect to Azure Data Lake Gen1. This allows users to reference data stored on your Azure Data Lake using the adl uri, for example: adl://myadl.azuredatalakestore.net.

  1. Create an Azure AD web application for service-to-service authentication. https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory

  2. Add the information from your web application to the hadoop core-site.xml configuration file:

<configuration>
        <property>
                <name>fs.adl.oauth2.access.token.provider.type</name>
                <value>ClientCredential</value>
        </property>
        <property>
                <name>fs.adl.oauth2.refresh.url</name>
                <value>Token endpoint created in step 1.</value>
        </property>
        <property>
                <name>fs.adl.oauth2.client.id</name>
                <value>Client ID created in step 1</value>
        </property>
        <property>
                <name>fs.adl.oauth2.credential</name>
                <value>Client Secret created in step 1</value>
        </property>
        <property>
                <name>fs.defaultFS</name>
                <value>ADL URIt</value>
        </property>
</configuration>

  1. Take note of the Hadoop Classpath and add the azure-datalake-store.jar file. This file can found on any hadoop version in: $HADOOP_HOME/share/hadoop/tools/lib/*

echo "$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*"

  1. Configure the Driverless AI config.toml file. Set the following configuration options:

enabled_file_systems = "upload, file, hdfs, azrbs, recipe_file, recipe_url"
hdfs_config_path = "/path/to/hadoop/conf"
hdfs_app_classpath = "/hadoop/classpath/"
hdfs_app_supported_schemes = "['adl://']"

  1. Save the changes when you are done, then stop/restart Driverless AI.

Example 4: Enable HDFS Connector to Connect to Azure Data Lake Gen 2

This example enables the HDFS Connector to connect to Azure Data Lake Gen2. This allows users to reference data stored on your Azure Data Lake using the Azure Blob Fyle System Driver, for example: abfs[s]://file_system@account_name.dfs.core.windows.net/<path>/<path>/<file_name>.

  1. Create an Azure Service Principal. https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

  2. Grant permissions to the Service Principal created on step 1 to access blobs: https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad

  3. Add the information from your web application to the hadoop core-site.xml configuration file:

<configuration>
        <property>
                <name>fs.azure.account.auth.type</name>
                <value>OAuth</value>
        </property>
        <property>
                <name>fs.azure.account.oauth.provider.type</name>
                <value>org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider</value>
        </property>
        <property>
                <name>fs.azure.account.oauth2.client.endpoint</name>
                <value>Token endpoint created in step 1.</value>
        </property>
        <property>
                <name>fs.azure.account.oauth2.client.id</name>
                <value>Client ID created in step 1</value>
        </property>
        <property>
                <name>fs.azure.account.oauth2.client.secret</name>
                <value>Client Secret created in step 1</value>
        </property>
</configuration>

  1. Take note of the Hadoop Classpath and add the required jar files. These files can found on any hadoop version 3.2 or higher at: $HADOOP_HOME/share/hadoop/tools/lib/*

echo "$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*"

Note: ABFS is only supported for hadoop version 3.2 or higher

  1. Configure the Driverless AI config.toml file. Set the following configuration options:

enabled_file_systems = "upload, file, hdfs, azrbs, recipe_file, recipe_url"
hdfs_config_path = "/path/to/hadoop/conf"
hdfs_app_classpath = "/hadoop/classpath/"
hdfs_app_supported_schemes = "['abfs://']"

  1. Save the changes when you are done, then stop/restart Driverless AI.

FAQ

Can I connect to my storage account using Private Endpoints?

Yes. Driverless AI can use privated endpoints if Driverless AI is located in the allowed VNET.

Does Driverless AI support secure transfer?

Yes. The Azure Blob Store Connector make all connections over HTTPS.

Does Driverless AI support hierarchical namespaces?

Yes.

Can I use Azure Managed Identities (MSI) to access the DataLake?

Yes. If Driverless AI is running on an Azure VM with managed identities. To enable the HDFS Connector to use MSI to authenticate, add to the core-site.xml:

For Gen1:

<property>
    <name>fs.adl.oauth2.access.token.provider.type</name>
    <value>MSI</value>
</property>

For Gen2:

<property>
    <name>fs.azure.account.auth.type</name>
    <value>OAuth</value>
</property>
<property>
    <name>fs.azure.account.oauth.provider.type</name>
    <value>org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider</value>
</property>