Use a Custom Scorer¶
First, we’ll initialize a client with our server credentials and store it in the variable dai
.
[1]:
import driverlessai
dai = driverlessai.Client(address='http://localhost:12345', username='py', password='py')
Here we grab a custom recipe from our recipe repo (https://github.com/h2oai/driverlessai-recipes) and upload it to the Driverless AI server.
[28]:
dai.recipes.create('https://github.com/h2oai/driverlessai-recipes/raw/rel-1.8.4.1/scorers/classification/multiclass/hamming_loss.py')
Complete 100%
It’s also possible to use the same dai.recipes.create()
function to upload recipes that we have written locally.
[29]:
dai.recipes.create('quadratic_weighted_kappa.py')
Complete 100%
We can create a list of custom scorer recipe objects.
[30]:
custom_scorers = [s for s in dai.recipes.scorers.list() if s.is_custom]
display(custom_scorers)
[<class 'driverlessai.recipes.ScorerRecipe'> HAMMING,
<class 'driverlessai.recipes.ScorerRecipe'> QWK]
For demonstration purposes, we’ll grab the first dataset available on the server. Then, we’ll use it to run an experiment that uses Hamming Loss for the scorer.
[31]:
ds = dai.datasets.list()[0]
ex = dai.experiments.create(
train_dataset=ds,
target_column=ds.columns[-1],
task='classification',
scorer=custom_scorers[0],
accuracy=1,
time=1
)
Experiment launched at: http://localhost:12345/#experiment?key=187cdc48-6004-11ea-919d-0242ac110002
Complete 100% - Status: Complete
We can see the scorer used by the experiment with the settings
attribute.
[32]:
ex.settings['scorer']
[32]:
'HAMMING'