Introduction to H2O AutoDoc

What is H2O AutoDoc

For many companies, model documentation is required for any model that the business uses. For other companies, model documentation is part of a data science team’s best practices. Model documentation includes how a model was created, the train and test dataset characteristics, how the model was evaluated, and what alternative models were considered. Collecting and documenting this information can take a data scientist days to complete for each model. The model document needs to be comprehensive and consistent across various projects. The process of creating this documentation, however, is tedious for the data scientist and wasteful for the business, because the data scientist could be using that time to build additional models and create more value. Furthermore, inconsistent or inaccurate model documentation can be an issue for model validation, governance, and regulatory compliance. H2O AutoDoc is designed to solve this problem.

H2O AutoDoc is a Python package that creates automatic reports for H2O-3 or scikit-learn supervised learning models.

Key Capabilities

• Automatic report generation in Microsoft Word (.docx) or Markdown (.md) formats

• Out-of-the-box report template included

• Template customization available to fit with your organization’s standards and requirements

• Support for models generated in H2O-3 and scikit-learn

Documentation Features

The editable report covers the model documentation spectrum including data overview, model results, and model interpretability among other critical details.

Key documentation sections include:

• Experiment Overview: provides an overview of the modeling problem

• System Specifications: describes the system configurations on which the model was trained

• Data Overview: summarizes the data shape and feature statistics

• Shifts Detected: highlights differences between train, validation, and test datasets

• Validation Strategy: summarizes how the validation dataset or datasets were created

• Model Parameters and Values: summarizes the model configuration and hyperparameters

• Performance Metrics: summarizes metrics relevant to the problem type

• Population Stability Index: highlights whether there is a shift in the predictions

• Prediction Statistics: summarizes prediction statistics for each dataset

• Feature Importance: includes model-specific feature importance

• Calibration Plots: summarizes how well the model is calibrated

• Partial Dependence Plots: provides insight into how a feature’s value can impact the target

• Alternative Models: summarizes alternative model results and configurations

Supported H2O-3 Models

• Deep Learning

• Distributed Random Forest (including Extremely Randomized Forest)

• Generalized Linear Model

• Gradient Boosting Machine

• Stacked Ensembles

• XGBoost

Supported Scikit-Learn Models

The AutoDoc supports most of scikit-learn’s supervised learning models for classification and regression. The AutoDoc, however, does not support the Multi-task algorithms.

Supported Software Details

• H2O-3

• Scikit-learn

H2O-3

AutoDoc for Binary vs. MOJO Models

The AutoDoc supports H2O-3 version 3.24.0.1 and higher. In version 3.24.0.3, H2O-3 added the ability to import MOJO models into an H2O-3 cluster. The MOJO import functionality provides a means to use external, pre-trained models in H2O - primarily for scoring. Depending on the external model type, select metrics and model information can also be attained. The AutoDoc leverages this MOJO import functionality to document MOJO models. Check the H2O-3 user guide to see if it supports your model’s MOJO here.

This table shows which H2O-3 versions AutoDoc supports for binary and MOJO models.

H2O-3 Version	Binary Model	MOJO Model
3.24.0.1 - 3.24.0.2	x
3.24.0.3 and greater	x	x

The following table shows when MOJO support was added for different models. Note: the AutoDoc includes all features for the supported H2O-3 binary models.

Notes:

• H2O-3 started supporting original model name in 3.30.0.1, in older versions you will see the model name as “Import MOJO Model”.

MOJO Model Type	Supported H2O Version
GBM	3.24.0.3 +
XGBoost	3.28.0.1 +
GLM	3.24.0.3 +
DRF	3.24.0.3 +
XRT	3.24.0.3 +

AutoDoc Sections by Problem-Type

Depending on your use case (i.e., classification or regression) the AutoDoc will exclude certain sections.

• Binary Classification - Includes all sections.

• Regression - Includes all sections except the classificaiton-specific sections:

  • Quantile Response Rate Plot

  • Actual vs Predicted Probabilities and Actual vs Predicted Log Odds

• Multiclass Classification - Includes all sections except:

  • Partial Dependence plots, which are not currently supported for multiclass problems

  • Population Stability Index

  • Prediction Statistics Table

  • Binary classification specific plots:

    • ROC curve

    • Cumulative Lift and Gains Charts

    • Quantile Response Rate Plot

    • Actual vs Predicted Probabilities and Actual vs Predicted Log Odds

Scikit-learn

The scikit-learn AutoDoc is designed to be very similar to the H2O-3 AutoDoc. In general, disparities between the AutoDocs are due to differences in H2O-3’s and scikit-learn’s algorithms.

Here we highlight some notable differences:

• Feature Importance is only provided for scikit-learn algorithms that support feature importance.

  • For scikit-learn 0.22 and higher, AutoDoc shows permutation importance for models that don’t have built-in feature importance.

• Validation Strategy section is only included in the H2O-3 AutoDoc.

  • H2O-3 estimators include hold-out data and cross-validation information, while scikit-learn estimators do not.

• Prediction Probability Information is limited to scikit-learn classifiers that provided prediction probabilities (accessible via the predict_proba() method). As a result, AutoDoc for classifiers without predict_probab() will exclude sections that require predicted probabilities:

  • Quantile Response Rates

  • Actual vs. Predicted, Prediction statistics

  • Population Stability Index.

  • Partial Dependence Plots

See Using H2O AutoDoc with Scikit-learn to know more.

Dataset Dependent Sections

The AutoDoc can render reports for models (Python Object, binary, MOJO, etc.,) with and without access to hold-out datasets. While H2O-3’s AutoDoc can render model-only reports, the scikit-learn AutoDoc requires that you pass in training data.

The following table summarizes AutoDocs with training data and/or hold-out datasets:

Section Topic	Model + Train	Model + Train + Hold-Out
Shift Detection	X
Population Stability Index	X
Hold-Out Data Metrics		X

If no datasets are provided, however, the report will exclude additional sections. The table below details the limitations of an H2O-3 model-only AutoDoc.

Section Name	Model and Data	Model
Experiment Overview	X	X
Data Overview	X
Validation Strategy	X
Feature Importance	X	Limited
Final Model	X	Limited
Alternative Models	X	X
Partial Dependence Plots	X
Appendix	X	Limited

Data Limitation Details by Section

Additional limitations can occur depending on the machine learning software platform (i.e., H2O-3, scikit-learn) and the software platform’s version number. See the corresponding software sections for further details.

• Feature Importance:

  • Excludes:

    • Shapley Feature Importance

• Final Model Section:

  • Excludes:

    • Population Stability Index

    • Prediction Statistics

    • Quantile Response Plots

    • Actual vs. Predicted Probabilities and Actual vs. Predicted Log Odds

  • Can Include:

    • Confusion Matrix

    • H2O-3 Performance Charts: Scoring History, ROC Curve, Cumulative Lift and Cumulative Gains

• Appendix:

  • Excludes:

    • Quantile Plot Calculation Tables (These tables only appear if quantile based table and plots are created)

Supported File Types

H2O AutoDoc can generate the following file types:

• Microsoft Word documents (.docx)

• Markdown files (.md)

Both of these file types can easily be exported to HTML or PDF.

Change Log

Version 1.0.5 (Feb 28, 2022)

• Support for H2O-3 version 3.34.0.3

• Ability to push generated AutoDoc to S3 Bucket and GitHub Repository

• Bug fixes and improvements on templates and documentation

Version 1.0.4 (Sep 17, 2021)

• Support for Python 3.8

• Bug fixes and improvements on templates and documentation

Version 1.0.3 (Nov 11, 2020)

• Documentation updates

Version 1.0.2 (August 27, 2020)

• New Features:

  • Allow Additional Testsets

  • Support Scikit-Learn Supervised Learning Models

  • Add AUCPR Plots

  • Add Cross-Validation Plots

  • Add Per Fold Cross-Validation Performance

• Improvements:

  • Return full path to the AutoDoc

  • Show All Final Model Parameters

• Bug Fixes:

  • Suppress Deprecation Warnings

  • Fix Key Error For Mojo with No Data

  • Fix Wrong “Generated On Date” for Mojo Models

Version 1.0.1 (July 30, 2020)

• Initial stable release