Automated Model Documentation (AutoDoc)

The following sections describe Driverless AI’s AutoDoc feature.

• Understanding AutoDoc

• Generating an AutoDoc

• Configuring AutoDoc

• Generating a Custom AutoDoc

Understanding AutoDoc

The AutoDoc feature is used to generate automated machine learning documentation for individual Driverless AI experiments. This editable document contains an overview of the experiment and includes other significant details like feature engineering and final model performance.

Custom AutoDocs

All Driverless AI experiments can generate either a standard or custom AutoDoc. A standard AutoDoc uses the default AutoDoc template that is included with Driverless AI, while a custom AutoDoc uses a customer-specific template that Driverless AI automatically populates.

If you are interested in creating a custom AutoDoc, contact support@h2o.ai. If you have already purchased a custom AutoDoc template and want to learn how to generate custom AutoDocs from your experiments, see Generating a Custom AutoDoc.

Generating an AutoDoc

Three different approaches can be used to generate an AutoDoc:

• Experiment UI

• MLI UI

• Python Client

Notes:

• For more information on how to configure plots/tables and enable/disable specific sections in the AutoDoc, see Configuring AutoDoc.

• These approaches also apply to custom AutoDocs. For more information, see Generating a Custom AutoDoc.

Experiment UI

Navigate to the Experiments page and click on the completed experiment you want to generate an AutoDoc for.

If AutoDoc was not previously enabled for the experiment, click the Build AutoDoc button.

If AutoDoc was previously enabled for the experiment, click the Download AutoDoc button.

MLI UI

Navigate to the MLI page and click on the completed experiment you want to generate an AutoDoc for.

Select AutoDoc from the MLI RECIPES’s menu and optionally select explainers that can be included in the AutoDoc (the standard AutoDoc supports the k-LIME Explainer and DT Surrogate Explainer).

The Standard AutoDoc with Explainers:

Python Client

• h2oai_client

• driverlessai

AutoDoc Functions

• create_and_download_autoreport()

• make_autodoc_sync()

For local downloads:

create_and_download_autoreport(
    model_key:str,
    template_path:str='',
    config_overrides:str='',
    dest_path:str='.',
    mli_key:str='',
    individual_rows:list=[],
    external_dataset_keys:list=[])

To save an AutoDoc to the DAI experiment directory (recommended if local downloads are disabled):

make_autodoc_sync(
    model_key:str,
    template_path:str='',
    config_overrides:str='',
    mli_key:str='',
    individual_rows:list=[],
    external_dataset_keys:list=[])

• model_key: The experiment key string.

• template_path: The full path the custom AutoDoc template.

• config_overrides: The TOML string format with configurations overrides for the AutoDoc.

• dest_path: The local path where the AutoDoc should be saved.

• mli_key: The mli key string.

• individual_rows: List of row indices for rows of interest in the training dataset, for which additional information can be shown (ICE, LOCO, KLIME).

• external_dataset_keys: List of DAI dataset keys.

h2oai_client

Connect to a running DAI instance:

from h2oai_client import Client
import driverlessai
address = 'http://ip_where_driverless_is_running:12345'
username = 'username'
password = 'password'
h2oai = Client(address=address, username=username, password=username)

Generate an AutoDoc and download it to your current working directory:

report = h2oai.create_and_download_autoreport(
    model_key=exp_key,
    dest_path:str='.',
)

driverlessai

Connect to a running DAI instance:

import driverlessai
address = 'http://ip_where_driverless_is_running:12345'
username = 'username'
password = 'password'
dai = driverlessai.Client(address=address, username=username, password=username)

Generate an AutoDoc and download it to your current working directory:

report = dai._backend.create_and_download_autoreport(
    model_key=exp_key,
    dest_path:str='.',
)

Configuring AutoDoc

An AutoDoc’s plots, tables, and sections can be configured through four different workflows:

• Experiment UI Expert Settings

• MLI UI Expert Settings

• Python Client

• config.toml file

Experiment UI Expert Settings

MLI UI Expert Settings

Python Client

• h2oai_client

• driverlessai

All configuration options for the AutoDoc are listed in the config.toml file. The following are several commonly used configuration parameters:

# Partial Dependence Plots (PDP) and ICE Plots
config_overrides = "autodoc_num_features=5"
config_overrides += "\nautodoc_pd_max_runtime=60"
config_overrides += "\nautodoc_num_rows=4"

# Prediction statistics
config_overrides += "\nautodoc_prediction_stats=true"
config_overrides += "\nautodoc_prediction_stats_n_quantiles=10"

# Population Stability Index (PSI)
config_overrides += "\nautodoc_population_stability_index=true"
config_overrides += "\nautodoc_population_stability_index_n_quantiles=10"

# Permutation feature importance
config_overrides += "\nautodoc_include_permutation_feature_importance=true"
config_overrides += "\nautodoc_feature_importance_scorer='GINI'"
config_overrides += "\nautodoc_feature_importance_num_perm=1"

# Response rates (only applicable to Binary classification)
config_overrides += "\nautodoc_response_rate=true"
config_overrides += "\nautodoc_response_rate_n_quantiles=10"

After setting these parameters, generate an AutoDoc and download it to your current working directory:

h2oai_client

report = h2oai.create_and_download_autoreport(
    model_key=exp_key,
    config_overrides=config_overrides,
    dest_path:str='.',
)

driverlessai

report = dai._backend.create_and_download_autoreport(
    model_key=exp_key,
    config_overrides=config_overrides,
    dest_path:str='.',
)

Generating a Custom AutoDoc

This section describes how to generate an AutoDoc from a custom AutoDoc template. Choose from the following options:

• Have Driverless AI use a custom AutoDoc for all experiments

• Have Driverless AI generate a custom AutoDoc for an individual experiment

Custom AutoDoc for All Experiments

Update the Config.toml file to point to the location of your custom AutoDoc.

Driverless AI must have access to the templates. Update the following path with your path.

autodoc_template='/full/path/to/your/custom_autodoc_template.docx'

Custom AutoDoc for Individual Experiments

You can use the Python Client to generate standard or custom AutoDocs from an experiment by setting the template_path variable to your custom AutoDoc’s path:

template_path='/full/path/to/your/custom_autodoc_template.docx'

Python Client: h2oai_client

report = h2oai.create_and_download_autoreport(
    model_key=exp_key,
    template_path=template_path,
    dest_path:str='.',
)

Python Client: driverlessai

report = dai._backend.create_and_download_autoreport(
    model_key=exp_key,
    template_path=template_path,
    dest_path:str='.',
)