Viewing Experiments

The upper-right corner of the Driverless AI UI includes an Experiments link.

Click this link to open the Experiments page. From this page, you can rename an experiment, view previous experiments, begin a new experiment, rerun an experiment, and delete an experiment.

Checkpointing, Rerunning, and Retraining

In Driverless AI, you can retry an experiment from the last checkpoint, you can run a new experiment using an existing experiment’s settings, and you can retrain an experiment’s final pipeline.

Checkpointing Experiments

In real-world scenarios, data can change. For example, you may have a model currently in production that was built using 1 million records. At a later date, you may receive several hundred thousand more records. Rather than building a new model from scratch, Driverless AI includes H2O.ai Brain, which enables caching and smart re-use of prior models to generate features for new models.

You can configure one of the following Brain levels in the experiment’s Expert Settings.

• -1: Don’t use any brain cache

• 0: Don’t use any brain cache but still write to cache

• 1: Smart checkpoint if an old experiment_id is passed in (for example, via running “resume one like this” in the GUI)

• 2: Smart checkpoint if the experiment matches all column names, column types, classes, class labels, and time series options identically. (default)

• 3: Smart checkpoint like level #1, but for the entire population. Tune only if the brain population is of insufficient size.

• 4: Smart checkpoint like level #2, but for the entire population. Tune only if the brain population is of insufficient size.

• 5: Smart checkpoint like level #4, but will scan over the entire brain cache of populations (starting from resumed experiment if chosen) in order to get the best scored individuals.

If you chooses Level 2 (default), then Level 1 is also done when appropriate.

To make use of smart checkpointing, be sure that the new data has:

• The same data column names as the old experiment

• The same data types for each column as the old experiment. (This won’t match if, e.g,. a column was all int and then had one string row.)

• The same target as the old experiment

• The same target classes (if classification) as the old experiment

• For time series, all choices for intervals and gaps must be the same

When the above conditions are met, then you can:

• Start the same kind of experiment, just rerun for longer.

• Use a smaller or larger data set (i.e. fewer or more rows).

• Effectively do a final ensemble re-fit by varying the data rows and starting an experiment with a new accuracy, time=1, and interpretability. Check the experiment preview for what the ensemble will be.

• Restart/Resume a cancelled, aborted, or completed experiment

To run smart checkpointing on an existing experiment, click the right side of the experiment that you want to retry, then select Restart from Last Checkpoint. The experiment settings page opens. Specify the new dataset. If desired, you can also change experiment settings, though the target column must be the same. Click Launch Experiment to resume the experiment from the last checkpoint and build a new experiment.

The smart checkpointing continues by adding a prior model as another model used during tuning. If that prior model is better (which is likely if it was run for more iterations), then that smart checkpoint model will be used during feature evolution iterations and final ensemble.

Notes:

• Driverless AI does not guarantee exact continuation, only smart continuation from any last point.

• The directory where the H2O.ai Brain meta model files are stored is tmp/H2O.ai_brain. In addition, the default maximum brain size is 20GB. Both the directory and the maximum size can be changed in the config.toml file.

Rerunning Experiments

To run a new experiment using an existing experiment’s settings, click the right side of the experiment that you want to use as the basis for the new experiment, then select New Model with Same Params. This opens the experiment settings page. From this page, you can rerun the experiment using the original settings, or you can specify to use new data and/or specify different experiment settings. Click Launch Experiment to create a new experiment with the same options.

Retrain Final Pipeline

To retrain an experiment’s final pipeline, click the right side of the experiment that you want to use as the basis for the new experiment, then select Retrain Final Pipeline. This opens the experiment settings page with the same settings as the original experiment except that Time is set to 0.

“Pausing” an Experiment

A trick for “pausing” an experiment is to:

1. Abort the experiment.

2. On the Experiments page, select Restart from Last Checkpoint for the aborted experiment.

3. On the Expert Settings page, specify 0 for the Ensemble Level for Final Modeling Pipeline option.

Deleting Experiments

To delete an experiment, click the right side of the experiment that you want to remove, then select Delete. A confirmation message will display asking you to confirm the delete. Click OK to delete the experiment or Cancel to return to the experiments page without deleting.