Leaderboards¶

Driverless AI provides a feature to automatically create leaderboards.

The Create Leaderboard feature runs multiple diverse experiments that provide an overview of the dataset. This feature also provides you with relevant information for deciding on complexity, accuracy, size, and time tradeoffs when putting models into production. Refer to the Expert Settings topic for information on expert settings that can be used to control this feature. For more information on the default models built for a leaderboard, see Leaderboard Models.

The built models are placed under the Project Workspace page and can be simultaneously scored on the test dataset and compared.

Creating a Leaderboard¶

Creating a Leaderboard is similar to running a new experiment. Refer to the Experiment Settings, Expert Settings, and Scorers topics for more information about options you can set when running an experiment.

On the Datasets page, select the dataset that you want to use for the experiment, then click Predict

or

On the Experiments page, click New Experiment, then select the dataset that you want to use.
Specify whether to include dropped columns, a validation dataset, and a testing dataset.
Specify the Target column and optionally a fold column, weight column, and time column.
Optionally specify Expert Settings.
Optionally adjust the Accuracy/Time/Interpretability knobs.
Optionally override the default scorer.
Optionally override the Classification/Regression setting.
Optionally specify to make the experiments reproducible and/or whether to enable GPUs.
Click the Create Leaderboard button.

Driverless AI creates a new, randomly named project and begins automatically training models using the queuing mechanism. The new project is given the description “Automatic Leader Board”. After all models have been built, you can score each experiment and compare experiments, as described in the Project Workspace topic.

Leaderboard Models¶

When creating a leaderboard, the models that are built will vary based on whether you are running a regular experiment or a time-series experiment.

Regular Experiments¶

By default, the following models are built when creating a leaderboard for regular (non-time-series) experiments. You can omit models from being built by disabling those models in the Expert Settings.

Model	Accuracy	Time	Interpretability	Config Overrides
Few Features Decision Tree	1	1	10	`max_orig_cols_selected=5` `nfeatures_max=10`
Simple LightGBM	1	1	10
Constant Baseline	1	1	10	`max_orig_cols_selected=1`
Single Decision Tree	Specified in experiment	Specified in experiment	Specified in experiment	`fixed_ensemble_level=0`
Single GLM	Specified in experiment	Specified in experiment	Specified in experiment	`fixed_ensemble_level=0`
Complex LightGBM Ensemble	7	Specified in experiment	Specified in experiment
Few Features Single LightGBM	Specified in experiment	Specified in experiment	Specified in experiment	`max_orig_cols_selected=5` `nfeatures_max=10` `fixed_ensemble_level=0`
Default Single LightGBM	Specified in experiment	Specified in experiment	Specified in experiment	`fixed_ensemble_level=0`
Default XGBoost/LightGBM Ensemble	Specified in experiment	Specified in experiment	Specified in experiment
Single FTRL	Specified in experiment	Specified in experiment	Specified in experiment	`fixed_ensemble_level=0`
Single TensorFlow	Specified in experiment	Specified in experiment	Specified in experiment	`fixed_ensemble_level=0`

Time Series Experiments¶

Driverless AI will build one time-series experiment using the default Driverless AI settings. The following additional models are also built when creating a leaderboard for time-series experiments.

You can control the time-series leaderboard with the time_series_leaderboard_mode TOML config. Specify one of the following options:

diverse: Explore a diverse set of models built using various expert settings. Note that you can rerun another diverse leaderboard on top of the best-performing model(s).
sliding_window: If the forecast horizon is N periods, create a separate model for each of the (gap, horizon) pairs of (0,n), (n,n), (2*n,n), …, (2*N-1, n) in units of time periods. The number of periods to predict per model n is controlled by the expert setting time_series_leaderboard_periods_per_model, which defaults to 1.

Notes:

Lag-based features are always included for time-series baseline models.
Unlike in regular experiments, the leaderboard will only build models using the specified Accuracy/Time/Interpretability settings.
For information about each transformer, see Driverless AI Transformations.

Model	Included Models	Included Transformers	Config Overrides
Exponential Smoothing	GLM	EwmaLagsTransformer	`nfeatures_max=10`
Target Lags DecisionTree	Decision Tree	LagsTransformer	`nfeatures_max=10`
Target Lags GLM	GLM	LagsTransformer, EwmaLagsTransformer, LagsAggregatesTransformer	`nfeatures_max=50`
Target Lags LightGBM/XGBoost	LightGBM, XGBoostGBM	LagsTransformer, EwmaLagsTransformer, LagsAggregatesTransformer	`nfeatures_max=50`