Using a Custom Scorer

Driverless AI supports a number of scorers, including:

• Regression: GINI, MAE, MAPE, MER, MSE, R2, RMSE (default), RMSLE, RMSPE, SMAPE, TOPDECILE

• Classification: ACCURACY, AUC (default), AUCPR, F05, F1, F2, GINI, LOGLOSS, MACROAUC, MCC

This example shows how you can include a custom scorer in your experiment. This example will use the Explained Variance scorer, which is used for regression experiments.

1. Start an experiment in Driverless AI by selecting your training dataset along with (optionally) validation and testing datasets and then specifying a (regression) Target Column.

2. The scorer defaults to RMSE. Click on Expert Settings.

3. Specify the custom scorer recipe using one of the following methods:

• On your local machine, clone the https://github.com/h2oai/driverlessai-recipes for this release branch. Then use the Upload Custom Recipe button to upload the driverlessai-recipes/scorers/explained_variance.py file.

• Click the Load Custom Recipe from URL button, then enter the URL for the raw h2o-3-models.py file (for example, https://raw.githubusercontent.com/h2oai/driverlessai-recipes/rel-1.8.10/scorers/regression/explained_variance.py).

Note: Click the Official Recipes (External) button to browse the driverlessai-recipes repository.

Driverless AI will begin uploading and verifying the new custom recipe.

4. In the Experiment Summary page, select the new Explained Variance (EXPVAR) scorer. (Note: If you do not see the EXPVAR option, return to the Expert Settings, select Recipes > Include Specific Scorers, then click the Enable Custom button in the top right corner. Click Done and then Save to return to the Experiment Summary.)

5. Edit any additional experiment settings, and then click Launch Experiment. The experiment will run using the custom Explained Variance scorer.