Enabling Notifications

Driverless AI can be configured to trigger a user-defined script at the beginning and end of an experiment. This functionality can be used to send notifications to services like Slack or to trigger a machine shutdown.

The config.toml file exposes the following variables:

  • listeners_experiment_start: Registers an absolute location of a script that gets executed at the start of an experiment.
  • listeners_experiment_done: Registers an absolute location of a script that gets executed when an experiment is finished.

Driverless AI accepts any executable as a script. (For example, a script can be implemented in Bash or Python.) There are only two requirements:

  • The specified script can be executed. (i.e., The file has executable flag.)
  • The script should be able to accept command line parameters.

Script Interfaces

When Driverless AI executes a script, it passes the following parameters as a script command line:

  • Application ID: A unique identifier of a running Driverless AI instance.
  • User ID: The identification of the user who is running the experiment.
  • Experiment ID: A unique identifier of the experiment.
  • Experiment Path: The location of the experiment results.

Examples

The following examples execute a given action when all launched experiments are finished. The examples vary slightly based on whether you are performing a Docker image install or a Native install. These examples use the following scripts:

  • The on_start Script. This script increases the counter of running experiments.

    #!/usr/bin/env bash
    
    app_id="${1}"
    experiment_id="${3}"
    tmp_dir="${TMPDIR:-/tmp}/${app_id}"
    exp_file="${tmp_dir}/${experiment_id}"
    
    mkdir -p "${tmp_dir}"
    touch "${exp_file}"
    
  • The on_done Script. This script decreases the counter and executes a specified action when the counter reaches 0-value.

    #!/usr/bin/env bash
    
    app_id="${1}"
    experiment_id="${3}"
    tmp_dir="${TMPDIR:-/tmp}/${app_id}"
    exp_file="${tmp_dir}/${experiment_id}"
    
    if [ -f "${exp_file}"  ]; then
        rm -f "${exp_file}"
    fi
    
    running_experiments=$(ls -1 "${tmp_dir}" | wc -l)
    
    if [ "${running_experiments}" -gt 0  ]; then
        echo "There is still ${running_experiments} running experiments!"
    else
        echo "No experiments running!"
    fi
    

Docker Image Users

  1. Copy the config.toml file from inside the Docker image to your local filesystem. (Change nvidia-docker run to docker run for non-GPU environments.)
# In your Driverless AI folder (for exmaple, dai_1.4.0),
# make config and scripts directories
mkdir config
mkdir scripts

# Copy the config.toml file to the new config directory.
nvidia-docker run \
  --pid=host \
  --rm \
  --init \
  -u `id -u`:`id -g` \
  -v `pwd`/config:/config \
  --entrypoint bash \
  h2oai/dai-centos7-x86_64:1.4.0-9.0
  -c "cp /etc/dai/config.toml /config"
  1. Edit the Notification scripts section in the config.toml file and save your changes. Note that in this example, the scripts are saved to a dai_1.4.0/scripts folder.
# Notification scripts
# - the variable points to a location of script which is executed at given event in experiment lifecycle
# - the script should have executable flag enabled
# - use of absolute path is suggested
# The on experiment start notification script location
listeners_experiment_start = "dai_1.4.0/scripts/on_start.sh"
# The on experiment finished notification script location
listeners_experiment_done = "dai_1.4.0/scripts/on_done.sh"
  1. Start Driverless AI with the DRIVERLESS_AI_CONFIG_FILE environment variable. Make sure this points to the location of the edited config.toml file so that the software finds the configuration file. (Change nvidia-docker run to docker run for non-GPU environments.)
nvidia-docker run \
  --pid=host \
  --init \
  --rm \
  -u `id -u`:`id -g` \
  -e DRIVERLESS_AI_CONFIG_FILE="/config/config.toml" \
  -v `pwd`/config:/config \
  -v `pwd`/data:/data \
  -v `pwd`/log:/log \
  -v `pwd`/license:/license \
  -v `pwd`/tmp:/tmp \
  -v `pwd`/scripts:/scripts \
  h2oai/dai-centos7-x86_64:1.4.0-9.0

Native Install Users

  1. Export the Driverless AI config.toml file or add it to ~/.bashrc. For example:
export DRIVERLESS_AI_CONFIG_FILE="/config/config.toml"
  1. Edit the Notification scripts section in the config.toml file to point to the new scripts. Save your changes when you are done.
# Notification scripts
# - the variable points to a location of script which is executed at given event in experiment lifecycle
# - the script should have executable flag enabled
# - use of absolute path is suggested
# The on experiment start notification script location
listeners_experiment_start = "/opt/h2oai/dai/scripts/on_start.sh"
# The on experiment finished notification script location
listeners_experiment_done = "/opt/h2oai/dai/scripts/on_done.sh"
  1. Start Driverless AI. Note that the command used to start Driverless AI varies depending on your install type.
# Deb or RPM with systemd (preferred for Deb and RPM):
# Start Driverless AI.
sudo systemctl start dai

# Deb or RPM without systemd:
# Start Driverless AI.
sudo -H -u dai /opt/h2oai/dai/run-dai.sh

# Tar.sh
# Start Driverless AI
./run-dai.sh