Using a Custom Scorer

Driverless AI supports a number of scorers, including:

  • Regression: GINI, MAE, MAPE, MER, MSE, R2, RMSE (default), RMSLE, RMSPE, SMAPE, TOPDECILE

  • Classification: ACCURACY, AUC (default), AUCPR, F05, F1, F2, GINI, LOGLOSS, MACROAUC, MCC

This example shows how you can include a custom scorer in your experiment. This example will use the Explained Variance scorer, which is used for regression experiments.

  1. Start an experiment in Driverless AI by selecting your training dataset along with (optionally) validation and testing datasets and then specifying a (regression) Target Column.

  2. The scorer defaults to RMSE. Click on Expert Settings.

  3. Specify the custom scorer recipe using one of the following methods:

  • On your local machine, clone the driverlessai-recipes repository, then use the Upload Custom Recipe button to upload the driverlessai-recipes/scorers/explained_variance.py file.

  • Click the Load Custom Recipe from URL button, then enter the URL for the Python file. (Both HTML and raw versions of the file are supported.)

Note: Click the Official Recipes (Open Source) button to browse the driverlessai-recipes repository.

Driverless AI will begin uploading and verifying the new custom recipe.

  1. In the Experiment Summary page, select the new Explained Variance (EXPVAR) scorer. (Note: If you do not see the EXPVAR option, return to the Expert Settings, select Recipes > Include Specific Scorers, then click the Enable Custom button in the top right corner. Click Done and then Save to return to the Experiment Summary.)

  2. Edit any additional experiment settings, and then click Launch Experiment. The experiment will run using the custom Explained Variance scorer.

Scorer Recipe example