Access Explainer generated data¶

Initialize a client with the server credentials and store them in the variable dai.

In [1]:

Copied!

import driverlessai

dai = driverlessai.Client(address='http://localhost:12345', username='py', password='py')
import driverlessai

dai = driverlessai.Client(address='http://localhost:12345', username='py', password='py')

Work with explainers¶

Select the creditcard binomial interpretation from Run an interpretation.

In [3]:

Copied!

dai.mli.iid.list()
dai.mli.iid.list()

Out[3]:

    | Type           | Key                                  | Name
----+----------------+--------------------------------------+-------------------------------
  0 | Interpretation | 85f3e0b6-1cf1-11ee-ac4c-ac1f6b643c68 | creditcard-mli-interpretation

In [4]:

Copied!

interpretation = dai.mli.iid.list()[0]
interpretation = dai.mli.iid.list()[0]

Access the explainers¶

Display a list of all the explainers that were executed.

In [5]:

Copied!

interpretation.explainers
interpretation.explainers

/usr/local/Caskroom/miniconda/base/envs/dai-py/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: 'Interpretation.explainers' is a beta API that is subject to future changes.
  """Entry point for launching an IPython kernel.

Out[5]:

	Key	Name
0	8669e9aa-1cf1-11ee-ac4c-ac1f6b643c68	Disparate Impact Analysis
1	8669fa8a-1cf1-11ee-ac4c-ac1f6b643c68	Sensitivity Analysis
2	866a026e-1cf1-11ee-ac4c-ac1f6b643c68	Shapley Values for Original Features (Naive Method)
3	866a0980-1cf1-11ee-ac4c-ac1f6b643c68	Shapley Summary Plot for Original Features (Naive Shapley Method)
4	866a102e-1cf1-11ee-ac4c-ac1f6b643c68	Shapley Values for Transformed Features
5	866a1696-1cf1-11ee-ac4c-ac1f6b643c68	Surrogate Decision Tree
6	866a1cb8-1cf1-11ee-ac4c-ac1f6b643c68	NLP Leave-one-covariate-out (LOCO)
7	866a22d0-1cf1-11ee-ac4c-ac1f6b643c68	NLP Partial Dependence Plot
8	866a28d4-1cf1-11ee-ac4c-ac1f6b643c68	Partial Dependence Plot

Get a reference to "Partial Dependence Plot" and "Shapley Values for Original Features (Naive Method)" explainers

In [6]:

Copied!

pdp = interpretation.explainers[8]
orig_shapley = interpretation.explainers[2]
pdp = interpretation.explainers[8]
orig_shapley = interpretation.explainers[2]

/usr/local/Caskroom/miniconda/base/envs/dai-py/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: 'Interpretation.explainers' is a beta API that is subject to future changes.
  """Entry point for launching an IPython kernel.
/usr/local/Caskroom/miniconda/base/envs/dai-py/lib/python3.7/site-packages/ipykernel_launcher.py:2: UserWarning: 'Interpretation.explainers' is a beta API that is subject to future changes.

In [8]:

Copied!

pdp.name
pdp.name

Out[8]:

'Partial Dependence Plot'

In [9]:

Copied!

pdp.id
pdp.id

Out[9]:

'h2oaicore.mli.byor.recipes.dai_pd_ice_explainer.DaiPdIceExplainer'

In [10]:

Copied!

orig_shapley.name
orig_shapley.name

Out[10]:

'Shapley Values for Original Features (Naive Method)'

In [11]:

Copied!

orig_shapley.id
orig_shapley.id

Out[11]:

'h2oaicore.mli.byor.recipes.original_contrib_explainer.NaiveShapleyExplainer'

Explainer Artifacts¶

The Explainer object provides the capability to download the explainer log and snapshot.

In [12]:

Copied!

pdp.artifacts
pdp.artifacts

Out[12]:

<class 'ExplainerArtifacts'> ['log', 'snapshot']

In [13]:

Copied!

orig_shapley.artifacts
orig_shapley.artifacts

Out[13]:

<class 'ExplainerArtifacts'> ['log', 'snapshot']

To download and view the explainer log, you can use the download_log() method provided by the Explainer object.

In [14]:

Copied!

pdp_artifacts = pdp.artifacts.download(only="log")
pdp_artifacts = pdp.artifacts.download(only="log")

In [15]:

Copied!

orig_shapley_artifacts = orig_shapley.artifacts.download(only="log", overwrite=True)
orig_shapley_artifacts = orig_shapley.artifacts.download(only="log", overwrite=True)

In [16]:

Copied!

print(pdp_artifacts)
print(pdp_artifacts)

{'log': 'explainer_run_866a28d4-1cf1-11ee-ac4c-ac1f6b643c68.log'}

In [17]:

Copied!

print(orig_shapley_artifacts)
print(orig_shapley_artifacts)

{'log': 'explainer_run_866a026e-1cf1-11ee-ac4c-ac1f6b643c68.log'}

Print the first 500 bytes of the log generated by the "Partial Dependence Plot" explainer.

In [18]:

Copied!

with open(pdp_artifacts['log']) as pdp_log:
    print(pdp_log.read(500))
with open(pdp_artifacts['log']) as pdp_log:
    print(pdp_log.read(500))

2023-07-07 18:10:18,910 C:  9% D:  NA    M:124.6GB NODE:SERVER      4034967 INFO    py       sys             | # Copyright 2020 H2O.ai; Proprietary License;  -*- encoding: utf-8 -*-
2023-07-07 18:10:18,910 C:  9% D:  NA    M:124.6GB NODE:SERVER      4034967 INFO    py       sys             | 
2023-07-07 18:10:18,910 C:  9% D:  NA    M:124.6GB NODE:SERVER      4034967 INFO    py       sys             | -------------------------------------------------------------------------------
2023-07-07 18:1

Explainer Frames¶

You can access the explainer frames generated by the 'Shapley Values for Original Features (Naive Method)' explainer.

In [19]:

Copied!

orig_shapley.frames
orig_shapley.frames

Out[19]:

<class 'ExplainerFrames'> ['shapley_orig']

In [20]:

Copied!

orig_shapley.frames.frame_names()
orig_shapley.frames.frame_names()

Out[20]:

['shapley_orig']

To access the explainer frames from the "Shapley Values for Original Features (Naive Method)" explainer, you can use orig_shapley.frames.frame_as_pandas() to open the frame as a pandas.DataFrame. Note that pandas must be installed to use this method. If it is not already installed, run pip install pandas to install the latest version.

In [21]:

Copied!

orig_shapley.frames.frame_as_pandas(frame_name="shapley_orig")
orig_shapley.frames.frame_as_pandas(frame_name="shapley_orig")

/usr/local/Caskroom/miniconda/base/envs/dai-py/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: 'ExplainerFrames.frame_as_pandas' is a beta API that is subject to future changes.
  """Entry point for launching an IPython kernel.

Out[21]:

	AGE	BILL_AMT1	BILL_AMT2	BILL_AMT3	BILL_AMT4	BILL_AMT5	BILL_AMT6	EDUCATION	LIMIT_BAL	MARRIAGE	...	PAY_5	PAY_6	PAY_AMT1	PAY_AMT2	PAY_AMT3	PAY_AMT4	PAY_AMT5	PAY_AMT6	SEX	bias
0	0.019977	0.200982	0.006168	-0.013152	0.009266	0.073089	0.023412	0.016237	-0.081920	0.023033	...	-0.070181	-0.048074	0.072944	-0.096812	0.033383	0.070340	-0.010721	0.004742	0.034740	-1.190962
1	-0.028986	0.024831	0.048321	0.070871	-0.026252	-0.004094	-0.011815	-0.017823	0.020453	-0.008035	...	-0.051930	0.227227	0.106323	0.103546	-0.039033	-0.008470	0.069028	-0.014369	-0.025161	-1.190962
2	0.001301	-0.076239	-0.020167	-0.003473	-0.000621	-0.001369	0.002048	0.004772	0.082133	-0.020433	...	-0.018518	0.000626	0.018723	0.114925	-0.044059	-0.026285	0.014725	0.008195	-0.003554	-1.190962
3	0.001692	-0.033329	-0.007122	0.005227	-0.001774	-0.002536	0.000000	0.022522	0.197826	0.025353	...	-0.023304	-0.005473	0.003756	-0.054085	-0.065979	-0.023279	0.004168	0.000753	-0.003554	-1.190962
4	0.005756	-0.068623	-0.026142	0.006067	-0.005933	-0.006837	-0.003336	0.022522	0.199707	0.022977	...	-0.024831	-0.005473	0.003756	-0.261130	-0.049136	-0.022065	0.018572	0.000753	0.006157	-1.190962
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
23994	-0.004668	-0.033329	-0.002249	0.002554	-0.001774	0.003588	0.000000	0.022522	0.285467	-0.017760	...	-0.022009	-0.003834	0.003756	-0.056469	-0.065979	-0.022065	-0.006887	0.013037	0.006157	-1.190962
23995	-0.064496	-0.016499	0.012413	0.035141	0.014207	-0.002979	0.026957	-0.019374	-0.000097	0.003898	...	-0.056840	-0.077220	-0.053675	0.042864	-0.028913	-0.126414	-0.011707	0.101891	0.018574	-1.190962
23996	-0.003833	-0.071306	-0.010470	0.004763	0.005318	-0.002536	-0.003580	0.016458	0.294947	0.025353	...	-0.021195	-0.002092	0.001100	-0.049379	-0.065979	0.000614	0.061146	-0.006323	0.000000	-1.190962
23997	-0.003833	-0.062491	-0.037640	0.003924	-0.003749	-0.003616	-0.003580	0.013971	0.275544	-0.013695	...	-0.021195	-0.001167	0.003904	-0.056469	-0.065979	0.018729	-0.032444	-0.002895	0.000000	-1.190962
23998	-0.003768	-0.059122	-0.014897	-0.009623	-0.002763	-0.001369	-0.033467	-0.104373	0.245434	-0.028363	...	-0.024250	-0.055067	0.006341	0.114179	0.079453	-0.032236	0.036079	0.002469	0.006157	-1.190962

23999 rows × 24 columns

The frames for the 'Shapley Values for Original Features (Naive Method)' explainer can also be downloaded directly using the orig_shapley.frames.download() method.

The pdp.frames is not available in the "Partial Dependence Plot" explainer, and will return None.

In [22]:

Copied!

pdp.frames is None
pdp.frames is None

Out[22]:

True

Explainer Data¶

To access the explainer data used, you can utilize the get_data() method.

Display the documentation for the get_data() method of the "Shapley Values for Original Features (Naive Method)" explainer.

In [24]:

Copied!

orig_shapley.get_data?
orig_shapley.get_data?

Signature: orig_shapley.get_data(*, clazz: 'str' = None)
Docstring:
Retrieve the ``ExplainerData`` from the Driverless AI server.
Raises a ``RuntimeError`` exception if the explainer has not been completed
successfully.

Keyword arguments
-----------------
clazz : str    [optional]
    Default: Selects the default or first class from the set of available classes in the
    multinomial classification model.
    The name of the class in multinomial classification
File:      /usr/local/Caskroom/miniconda/base/envs/dai-py/lib/python3.7/site-packages/driverlessai/_mli.py
Type:      method

You can use this method to obtain the explainer data for the default class.

In [25]:

Copied!

orig_shapley_data = orig_shapley.get_data()
orig_shapley_data
orig_shapley_data = orig_shapley.get_data()
orig_shapley_data

Out[25]:

<class 'ExplainerData'> global-feature-importance

The raw JSON string can be accessed via the data property of orig_shapley_data.

In [26]:

Copied!

orig_shapley_data.data
orig_shapley_data.data

Out[26]:

'[{"label": "PAY_0", "value": 0.4113123313148533, "scope": "global"}, {"label": "PAY_2", "value": 0.13670637041425462, "scope": "global"}, {"label": "LIMIT_BAL", "value": 0.10956191060352012, "scope": "global"}, {"label": "PAY_AMT2", "value": 0.0858049774208528, "scope": "global"}, {"label": "BILL_AMT1", "value": 0.076513771503827, "scope": "global"}, {"label": "PAY_AMT3", "value": 0.0738815581187556, "scope": "global"}, {"label": "PAY_3", "value": 0.0716208421239144, "scope": "global"}, {"label": "PAY_AMT1", "value": 0.049943364745309145, "scope": "global"}, {"label": "PAY_AMT4", "value": 0.044986743259307335, "scope": "global"}, {"label": "PAY_5", "value": 0.04138120606988586, "scope": "global"}, {"label": "BILL_AMT2", "value": 0.035286053930605146, "scope": "global"}, {"label": "PAY_4", "value": 0.0329864470420603, "scope": "global"}, {"label": "PAY_AMT5", "value": 0.032496230489983374, "scope": "global"}, {"label": "PAY_6", "value": 0.024884367764080827, "scope": "global"}, {"label": "PAY_AMT6", "value": 0.02419174204061356, "scope": "global"}, {"label": "AGE", "value": 0.022399574038290702, "scope": "global"}, {"label": "EDUCATION", "value": 0.01929306981437939, "scope": "global"}, {"label": "BILL_AMT4", "value": 0.016303021034977053, "scope": "global"}, {"label": "MARRIAGE", "value": 0.0158217246555915, "scope": "global"}, {"label": "BILL_AMT6", "value": 0.015482189541597725, "scope": "global"}, {"label": "BILL_AMT3", "value": 0.013910526069429722, "scope": "global"}, {"label": "BILL_AMT5", "value": 0.011513149153218907, "scope": "global"}, {"label": "SEX", "value": 0.007060633089822252, "scope": "global"}]'

You can view the data as a Python data structure. To access the data as a Python data structure, you can use the orig_shapley_data.data_as_dict() method.

In [27]:

Copied!

orig_shapley_data.data_as_dict()
orig_shapley_data.data_as_dict()

Out[27]:

[{'label': 'PAY_0', 'value': 0.4113123313148533, 'scope': 'global'},
 {'label': 'PAY_2', 'value': 0.13670637041425462, 'scope': 'global'},
 {'label': 'LIMIT_BAL', 'value': 0.10956191060352012, 'scope': 'global'},
 {'label': 'PAY_AMT2', 'value': 0.0858049774208528, 'scope': 'global'},
 {'label': 'BILL_AMT1', 'value': 0.076513771503827, 'scope': 'global'},
 {'label': 'PAY_AMT3', 'value': 0.0738815581187556, 'scope': 'global'},
 {'label': 'PAY_3', 'value': 0.0716208421239144, 'scope': 'global'},
 {'label': 'PAY_AMT1', 'value': 0.049943364745309145, 'scope': 'global'},
 {'label': 'PAY_AMT4', 'value': 0.044986743259307335, 'scope': 'global'},
 {'label': 'PAY_5', 'value': 0.04138120606988586, 'scope': 'global'},
 {'label': 'BILL_AMT2', 'value': 0.035286053930605146, 'scope': 'global'},
 {'label': 'PAY_4', 'value': 0.0329864470420603, 'scope': 'global'},
 {'label': 'PAY_AMT5', 'value': 0.032496230489983374, 'scope': 'global'},
 {'label': 'PAY_6', 'value': 0.024884367764080827, 'scope': 'global'},
 {'label': 'PAY_AMT6', 'value': 0.02419174204061356, 'scope': 'global'},
 {'label': 'AGE', 'value': 0.022399574038290702, 'scope': 'global'},
 {'label': 'EDUCATION', 'value': 0.01929306981437939, 'scope': 'global'},
 {'label': 'BILL_AMT4', 'value': 0.016303021034977053, 'scope': 'global'},
 {'label': 'MARRIAGE', 'value': 0.0158217246555915, 'scope': 'global'},
 {'label': 'BILL_AMT6', 'value': 0.015482189541597725, 'scope': 'global'},
 {'label': 'BILL_AMT3', 'value': 0.013910526069429722, 'scope': 'global'},
 {'label': 'BILL_AMT5', 'value': 0.011513149153218907, 'scope': 'global'},
 {'label': 'SEX', 'value': 0.007060633089822252, 'scope': 'global'}]

You can also view the data as a pandas.DataFrame using orig_shapley_data.data_as_pandas(). Note that this requires the pandas library to be installed. If pandas is not already installed, you can use pip install pandas to install the latest version.

In [28]:

Copied!

orig_shapley_data.data_as_pandas()
orig_shapley_data.data_as_pandas()

/usr/local/Caskroom/miniconda/base/envs/dai-py/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: 'ExplainerData.data_as_pandas' is a beta API that is subject to future changes.
  """Entry point for launching an IPython kernel.

Out[28]:

	label	value	scope
0	PAY_0	0.411312	global
1	PAY_2	0.136706	global
2	LIMIT_BAL	0.109562	global
3	PAY_AMT2	0.085805	global
4	BILL_AMT1	0.076514	global
5	PAY_AMT3	0.073882	global
6	PAY_3	0.071621	global
7	PAY_AMT1	0.049943	global
8	PAY_AMT4	0.044987	global
9	PAY_5	0.041381	global
10	BILL_AMT2	0.035286	global
11	PAY_4	0.032986	global
12	PAY_AMT5	0.032496	global
13	PAY_6	0.024884	global
14	PAY_AMT6	0.024192	global
15	AGE	0.022400	global
16	EDUCATION	0.019293	global
17	BILL_AMT4	0.016303	global
18	MARRIAGE	0.015822	global
19	BILL_AMT6	0.015482	global
20	BILL_AMT3	0.013911	global
21	BILL_AMT5	0.011513	global
22	SEX	0.007061	global

Display the documentation for the get_data() method of the "Partial Dependence Plot" explainer.

In [29]:

Copied!

pdp.get_data?
pdp.get_data?

Signature: pdp.get_data(*, feature_name: 'str', clazz: 'str' = None)
Docstring:
Retrieve the ``ExplainerData`` from the Driverless AI server.
Raises a ``RuntimeError`` exception if the explainer has not been completed
successfully.

Keyword arguments
-----------------
feature_name : str    [required]
    The name of the feature whose data we want to retrieve.
clazz : str    [optional]
    Default: Selects the default or first class from the set of available classes in the
    multinomial classification model.
    The name of the class in multinomial classification
File:      /usr/local/Caskroom/miniconda/base/envs/dai-py/lib/python3.7/site-packages/driverlessai/_mli.py
Type:      method

To use the pdp.get_data() method, the feature_name keyword argument is required. In this case, the value "PAY_2" is passed.

In [30]:

Copied!

pdp_data = pdp.get_data(feature_name="PAY_2")
pdp_data
pdp_data = pdp.get_data(feature_name="PAY_2")
pdp_data

Out[30]:

<class 'ExplainerData'> global-partial-dependence

To display the "Partial Dependence Plot" explainer data as a pandas.DataFrame, you can use the pdp_data.data_as_pandas() method.

In [31]:

Copied!

pdp_data.data_as_pandas()
pdp_data.data_as_pandas()

/usr/local/Caskroom/miniconda/base/envs/dai-py/lib/python3.7/site-packages/ipykernel_launcher.py:1: UserWarning: 'ExplainerData.data_as_pandas' is a beta API that is subject to future changes.
  """Entry point for launching an IPython kernel.

Out[31]:

	bin	pd	sd	oor
0	-2	0.218255	0.182037	False
1	-1	0.217817	0.181967	False
2	0	0.218528	0.183518	False
3	1	0.233411	0.175506	False
4	2	0.314469	0.176374	False
5	3	0.309362	0.169429	False
6	4	0.301135	0.161080	False
7	5	0.303885	0.165119	False
8	6	0.304114	0.165656	False
9	7	0.302250	0.163217	False
10	8	0.302250	0.163217	False

In [ ]: