Model Diagnostics¶

This page describes the admin API, which lets you manage administrative operations in the Driverless AI Python client.

In [5]:

import driverlessai
client = driverlessai.Client(address='http://localhost:12345', username="py", password="py")

import driverlessai client = driverlessai.Client(address='http://localhost:12345', username="py", password="py")

Create a model diagnostic and store it in the variable model_diagnostic

In [12]:

model_diagnostic = client.model_diagnostics.create(
        experiment,
        test_dataset,
    )

model_diagnostic = client.model_diagnostics.create( experiment, test_dataset, )

Complete 100.00% - Click to open diagnostic

Experiment¶

You can retrieve the experiment used for the model diagnostic as follows.

In [11]:

model_diagnostic.experiment

model_diagnostic.experiment

Out[11]:

<class 'Experiment'> 5adf2a9a-e9f0-11ed-b1d1-ac1f6b6b4a0e experiment_classification

Test Dataset¶

You can also retrieve the test dataset used for the model diagnostic.

In [12]:

model_diagnostic.test_dataset

model_diagnostic.test_dataset

Out[12]:

<class 'Dataset'> 1b25159a-e9f0-11ed-b1d1-ac1f6b6b4a0e test_dataset

Scores¶

The scores of the model diagnostic can be retrieved as follows.

In [13]:

model_diagnostic.scores

model_diagnostic.scores

Out[13]:

{'ACCURACY': {'score': 0.835,
  'mean': 0.835961111111111,
  'sd': 0.0038955538125114903},
 'AUC': {'score': 0.7974402510560225,
  'mean': 0.7971412843174919,
  'sd': 0.006710092535550581},
 'AUCPR': {'score': 0.5762736392953374,
  'mean': 0.568546740239218,
  'sd': 0.012683703591743376},
 'F05': {'score': 0.5995985950827898,
  'mean': 0.5998075359459527,
  'sd': 0.01558847749567046},
 'F1': {'score': 0.555921052631579,
  'mean': 0.5590555375755546,
  'sd': 0.010589199199421487},
 'F2': {'score': 0.6402439024390244,
  'mean': 0.6410757215697164,
  'sd': 0.008460965789798264},
 'FDR': {'score': 0.4202401372212693,
  'mean': 0.43331110845581083,
  'sd': 0.031438121998583715},
 'FNR': {'score': 0.4660347551342812,
  'mean': 0.4463239190632311,
  'sd': 0.03092405588998729},
 'FOR': {'score': 0.1220521307405875,
  'mean': 0.12084346186678073,
  'sd': 0.0044743293095448056},
 'FPR': {'score': 0.10350654837346852,
  'mean': 0.11233871711746916,
  'sd': 0.01720356137045864},
 'GINI': {'score': 0.5948805021120449,
  'mean': 0.6020276982006425,
  'sd': 0.012508107202368652},
 'LOGLOSS': {'score': 0.40756278844550253,
  'mean': 0.4092149926647026,
  'sd': 0.00806472641732421},
 'MACROAUC': {'score': 0.7974402510560225,
  'mean': 0.7973613657682173,
  'sd': 0.0062434380925958126},
 'MACROF1': {'score': 0.555921052631579,
  'mean': 0.5589283108692615,
  'sd': 0.013952122545669392},
 'MACROMCC': {'score': 0.4442232496953473,
  'mean': 0.4444346144104687,
  'sd': 0.014543630523048617},
 'MCC': {'score': 0.4442232496953473,
  'mean': 0.4505255617689142,
  'sd': 0.015409575236966358},
 'NPV': {'score': 0.8779478692594125,
  'mean': 0.8810014209808824,
  'sd': 0.0048437028400385145},
 'PRECISION': {'score': 0.5797598627787307,
  'mean': 0.5751486714549728,
  'sd': 0.021209317777230865},
 'RECALL': {'score': 0.5339652448657188,
  'mean': 0.5492359241441241,
  'sd': 0.02093284195977224},
 'TNR': {'score': 0.8964934516265315,
  'mean': 0.8828648984221935,
  'sd': 0.024396342565919823}}

Metric Plots¶

All the metric plots of the model diagnostic can be accessed as follows.

In [13]:

metric_plots = model_diagnostic.metric_plots

metric_plots = model_diagnostic.metric_plots

/var/folders/y8/4wlw8c15491g6n1xh8tfzd700000gn/T/ipykernel_81065/1450864514.py:1: UserWarning: 'ModelDiagnostic.metric_plots' is a beta API that is subject to future changes.
  metric_plots = model_diagnostic.metric_plots

This returns an instance of the MetricPlots class which can be used to retrieve each plot. Each graph is rendered in Vega Lite(3) format. For more information, see https://vega.github.io/vega-lite-v3/.

In [15]:

from vega import Vega

from vega import Vega

Actual Vs Predicted Chart¶

In [20]:

Vega(metric_plots.actual_vs_predicted_chart)

Vega(metric_plots.actual_vs_predicted_chart)

Out[20]:

Cumulative Gain Chart¶

In [21]:

Vega(metric_plots.gains_chart)

Vega(metric_plots.gains_chart)

Out[21]:

Kolmogorov-Smirnov Chart¶

In [22]:

Vega(metric_plots.ks_chart)

Vega(metric_plots.ks_chart)

Out[22]:

Lift Chart¶

In [23]:

Vega(metric_plots.lift_chart)

Vega(metric_plots.lift_chart)

Out[23]:

Precision-Recall Curve¶

In [24]:

Vega(metric_plots.prec_recall_curve)

Vega(metric_plots.prec_recall_curve)

Out[24]:

Residual Histogram¶

In [25]:

Vega(metric_plots.residual_histogram)

Vega(metric_plots.residual_histogram)

Out[25]:

Residual Plot¶

In [26]:

Vega(metric_plots.residual_plot)

Vega(metric_plots.residual_plot)

Out[26]:

ROC Curve¶

In [27]:

Vega(metric_plots.roc_curve)

Vega(metric_plots.roc_curve)

Out[27]:

Confusion Matrix¶

In [30]:

metric_plots.confusion_matrix(threshold=0.6)

metric_plots.confusion_matrix(threshold=0.6)

Out[30]:

[['', '0', '1', 'Total', 'Error'],
 ['0', 4672, 62, 4734.0, 0.0131],
 ['1', 1073, 193, 1266.0, 0.8476],
 ['Total', 5745.0, 255.0, 6000.0],
 ['Error', 0.1868, 0.2431, '', 0.1892]]

Please consider that, when a certain plot is not available for a particular model diagnostic, it will return None.

In [33]:

metric_plots.residual_plot

metric_plots.residual_plot