Use a Custom Scorer¶

First, we'll initialize a client with our server credentials and store it in the variable dai.

In [1]:

import driverlessai

dai = driverlessai.Client(address='http://localhost:12345', username='py', password='py')

import driverlessai dai = driverlessai.Client(address='http://localhost:12345', username='py', password='py')

Here we grab a custom recipe from our recipe repo (https://github.com/h2oai/driverlessai-recipes) and upload it to the Driverless AI server.

In [28]:

dai.recipes.create('https://github.com/h2oai/driverlessai-recipes/blob/master/scorers/classification/multiclass/hamming_loss.py')

dai.recipes.create('https://github.com/h2oai/driverlessai-recipes/blob/master/scorers/classification/multiclass/hamming_loss.py')

Complete 100%

It's also possible to use the same dai.recipes.create() function to upload recipes that we have written locally.

In [29]:

dai.recipes.create('quadratic_weighted_kappa.py')

dai.recipes.create('quadratic_weighted_kappa.py')

Complete 100%

We can create a list of custom scorer recipe objects.

In [30]:

custom_scorers = [s for s in dai.recipes.scorers.list() if s.is_custom]
display(custom_scorers)

custom_scorers = [s for s in dai.recipes.scorers.list() if s.is_custom] display(custom_scorers)

[<class 'driverlessai.recipes.ScorerRecipe'> HAMMING,
 <class 'driverlessai.recipes.ScorerRecipe'> QWK]

For demonstration purposes, we'll grab the first dataset available on the server. Then, we'll use it to run an experiment that uses Hamming Loss for the scorer.

In [31]:

ds = dai.datasets.list()[0]
ex = dai.experiments.create(
    train_dataset=ds, 
    target_column=ds.columns[-1], 
    task='classification', 
    scorer=custom_scorers[0],
    accuracy=1,
    time=1
)

ds = dai.datasets.list()[0] ex = dai.experiments.create( train_dataset=ds, target_column=ds.columns[-1], task='classification', scorer=custom_scorers[0], accuracy=1, time=1 )

Experiment launched at: http://localhost:12345/#experiment?key=187cdc48-6004-11ea-919d-0242ac110002
Complete 100% - Status: Complete                                                  

We can see the scorer used by the experiment with the settings attribute.

In [32]:

ex.settings['scorer']

ex.settings['scorer']

Out[32]:

'HAMMING'