Leaderboards

Driverless AI provides a feature to automatically create leaderboards.

The Create Leaderboard feature runs multiple diverse experiments that provide an overview of the dataset. This feature also provides you with relevant information for deciding on complexity, accuracy, size, and time tradeoffs when putting models into production. Refer to the Expert Settings topic for information on expert settings that can be used to control this feature. For more information on the default models built for a leaderboard, see Leaderboard Models.

The built models are placed under the Project Workspace page and can be simultaneously scored on the test dataset and compared.

Creating a Leaderboard

Creating a Leaderboard is similar to running a new experiment. Refer to the Experiment Settings, Expert Settings, and Scorers topics for more information about options you can set when running an experiment.

  1. On the Datasets page, select the dataset that you want to use for the experiment, then click Predict

    or

    On the Experiments page, click New Experiment, then select the dataset that you want to use.

  2. Specify whether to include dropped columns, a validation dataset, and a testing dataset.

  3. Specify the Target column and optionally a fold column, weight column, and time column.

  4. Optionally specify Expert Settings.

  5. Optionally adjust the Accuracy/Time/Interpretability knobs.

  6. Optionally override the default scorer.

  7. Optionally override the Classification/Regression setting.

  8. Optionally specify to make the experiments reproducible and/or whether to enable GPUs.

  9. Click the Create Leaderboard button.

Create Leaderboard

Driverless AI creates a new, randomly named project and begins automatically training models using the queuing mechanism. The new project is given the description “Automatic Leader Board”. After all models have been built, you can score each experiment and compare experiments, as described in the Project Workspace topic.

Projects page for leaderboard

Leaderboard Models

When creating a leaderboard, the models that are built will vary based on whether you are running a regular experiment or a time-series experiment.

Regular Experiments

By default, the following models are built when creating a leaderboard for regular (non-time-series) experiments. You can omit models from being built by disabling those models in the Expert Settings.

Model

Accuracy

Time

Interpretability

Config Overrides

Few Features Decision Tree

1

1

10

max_orig_cols_selected=5 nfeatures_max=10

Simple LightGBM

1

1

10

Constant Baseline

1

1

10

max_orig_cols_selected=1

Single Decision Tree

Specified in experiment

Specified in experiment

Specified in experiment

fixed_ensemble_level=0

Single GLM

Specified in experiment

Specified in experiment

Specified in experiment

fixed_ensemble_level=0

Complex LightGBM Ensemble

7

Specified in experiment

Specified in experiment

Few Features Single LightGBM

Specified in experiment

Specified in experiment

Specified in experiment

max_orig_cols_selected=5 nfeatures_max=10 fixed_ensemble_level=0

Default Single LightGBM

Specified in experiment

Specified in experiment

Specified in experiment

fixed_ensemble_level=0

Default XGBoost/LightGBM Ensemble

Specified in experiment

Specified in experiment

Specified in experiment

Single FTRL

Specified in experiment

Specified in experiment

Specified in experiment

fixed_ensemble_level=0

Single TensorFlow

Specified in experiment

Specified in experiment

Specified in experiment

fixed_ensemble_level=0

Time Series Experiments

Driverless AI will build one time-series experiment using the default Driverless AI settings. The following additional models are also built when creating a leaderboard for time-series experiments.

You can control the time-series leaderboard with the time_series_leaderboard_mode TOML config. Specify one of the following options:

  • diverse: Explore a diverse set of models built using various expert settings. Note that you can rerun another diverse leaderboard on top of the best-performing model(s).

  • sliding_window: If the forecast horizon is N periods, create a separate model for each of the (gap, horizon) pairs of (0,n), (n,n), (2*n,n), …, (2*N-1, n) in units of time periods. The number of periods to predict per model n is controlled by the expert setting time_series_leaderboard_periods_per_model, which defaults to 1.

Notes:

  • Lag-based features are always included for time-series baseline models.

  • Unlike in regular experiments, the leaderboard will only build models using the specified Accuracy/Time/Interpretability settings.

  • For information about each transformer, see Driverless AI Transformations.

Model

Included Models

Included Transformers

Config Overrides

Exponential Smoothing

GLM

EwmaLagsTransformer

nfeatures_max=10

Target Lags DecisionTree

Decision Tree

LagsTransformer

nfeatures_max=10

Target Lags GLM

GLM

LagsTransformer, EwmaLagsTransformer, LagsAggregatesTransformer

nfeatures_max=50

Target Lags LightGBM/XGBoost

LightGBM, XGBoostGBM

LagsTransformer, EwmaLagsTransformer, LagsAggregatesTransformer

nfeatures_max=50