Use a Custom Scorer

First, we’ll initialize a client with our server credentials and store it in the variable dai.

import driverlessai

dai = driverlessai.Client(address='http://localhost:12345', username='py', password='py')

Here we grab a custom recipe from our recipe repo (https://github.com/h2oai/driverlessai-recipes) and upload it to the Driverless AI server.

dai.recipes.create('https://github.com/h2oai/driverlessai-recipes/raw/rel-1.8.4.1/scorers/classification/multiclass/hamming_loss.py')

Complete 100%

It’s also possible to use the same dai.recipes.create() function to upload recipes that we have written locally.

dai.recipes.create('quadratic_weighted_kappa.py')

Complete 100%

We can create a list of custom scorer recipe objects.

custom_scorers = [s for s in dai.recipes.scorers.list() if s.is_custom]
display(custom_scorers)

[<class 'driverlessai.recipes.ScorerRecipe'> HAMMING,
 <class 'driverlessai.recipes.ScorerRecipe'> QWK]

For demonstration purposes, we’ll grab the first dataset available on the server. Then, we’ll use it to run an experiment that uses Hamming Loss for the scorer.

ds = dai.datasets.list()[0]
ex = dai.experiments.create(
    train_dataset=ds,
    target_column=ds.columns[-1],
    task='classification',
    scorer=custom_scorers[0],
    accuracy=1,
    time=1
)

Experiment launched at: http://localhost:12345/#experiment?key=187cdc48-6004-11ea-919d-0242ac110002
Complete 100% - Status: Complete

We can see the scorer used by the experiment with the settings attribute.

ex.settings['scorer']

'HAMMING'