Introduction to H2O AutoDoc =========================== What is H2O AutoDoc ------------------- For many companies, model documentation is required for any model that the business uses. For other companies, model documentation is part of a data science team’s best practices. Model documentation includes how a model was created, the train and test dataset characteristics, how the model was evaluated, and what alternative models were considered. Collecting and documenting this information can take a data scientist days to complete for each model. The model document needs to be comprehensive and consistent across various projects. The process of creating this documentation, however, is tedious for the data scientist and wasteful for the business, because the data scientist could be using that time to build additional models and create more value. Furthermore, inconsistent or inaccurate model documentation can be an issue for model validation, governance, and regulatory compliance. H2O AutoDoc is designed to solve this problem. H2O AutoDoc is a Python package that creates automatic reports for H2O-3 or scikit-learn supervised learning models. Key Capabilities ~~~~~~~~~~~~~~~~ * Automatic report generation in Microsoft Word (.docx) or Markdown (.md) formats * Out-of-the-box report template included * Template customization available to fit with your organization’s standards and requirements * Support for models generated in H2O-3 and scikit-learn Documentation Features ~~~~~~~~~~~~~~~~~~~~~~ The editable report covers the model documentation spectrum including data overview, model results, and model interpretability among other critical details. Key documentation sections include: * **Experiment Overview**: provides an overview of the modeling problem * **System Specifications**: describes the system configurations on which the model was trained * **Data Overview**: summarizes the data shape and feature statistics * **Shifts Detected**: highlights differences between train, validation, and test datasets * **Validation Strategy**: summarizes how the validation dataset or datasets were created * **Model Parameters and Values**: summarizes the model configuration and hyperparameters * **Performance Metrics**: summarizes metrics relevant to the problem type * **Population Stability Index**: highlights whether there is a shift in the predictions * **Prediction Statistics**: summarizes prediction statistics for each dataset * **Feature Importance**: includes model-specific feature importance * **Calibration Plots**: summarizes how well the model is calibrated * **Partial Dependence Plots**: provides insight into how a feature's value can impact the target * **Alternative Models**: summarizes alternative model results and configurations Supported H2O-3 Models ~~~~~~~~~~~~~~~~~~~~~~ * Deep Learning * Distributed Random Forest (including Extremely Randomized Forest) * Generalized Linear Model * Gradient Boosting Machine * Stacked Ensembles * XGBoost Supported Scikit-Learn Models ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The AutoDoc supports most of scikit-learn's supervised learning models for classification and regression. The AutoDoc, however, does not support the Multi-task algorithms. Supported Software Details --------------------------- - H2O-3_ - Scikit-learn_ .. _H2O-3: H2O-3 ~~~~~~ **AutoDoc for Binary vs. MOJO Models** The AutoDoc supports H2O-3 version 3.24.0.1 and higher. In version 3.24.0.3, H2O-3 added the ability to import MOJO models into an H2O-3 cluster. The MOJO import functionality provides a means to use external, pre-trained models in H2O - primarily for scoring. Depending on the external model type, select metrics and model information can also be attained. The AutoDoc leverages this MOJO import functionality to document MOJO models. Check the H2O-3 user guide to see if it supports your model's MOJO `here `_. This table shows which H2O-3 versions AutoDoc supports for binary and MOJO models. +---------------------+--------------+------------+ | H2O-3 Version | Binary Model | MOJO Model | +=====================+==============+============+ | 3.24.0.1 - 3.24.0.2 | x | | +---------------------+--------------+------------+ | 3.24.0.3 and greater| x | x | +---------------------+--------------+------------+ The following table shows when MOJO support was added for different models. Note: the AutoDoc includes all features for the supported H2O-3 binary models. Notes: - H2O-3 started supporting original model name in 3.30.0.1, in older versions you will see the model name as "Import MOJO Model". +-----------------+----------------------+ | MOJO Model Type | Supported H2O Version| +=================+======================+ | GBM | 3.24.0.3 + | +-----------------+----------------------+ | XGBoost | 3.28.0.1 + | +-----------------+----------------------+ | GLM | 3.24.0.3 + | +-----------------+----------------------+ | DRF | 3.24.0.3 + | +-----------------+----------------------+ | XRT | 3.24.0.3 + | +-----------------+----------------------+ **AutoDoc Sections by Problem-Type** Depending on your use case (i.e., classification or regression) the AutoDoc will exclude certain sections. * Binary Classification - Includes all sections. * Regression - Includes all sections except the classificaiton-specific sections: * Quantile Response Rate Plot * Actual vs Predicted Probabilities and Actual vs Predicted Log Odds * Multiclass Classification - Includes all sections except: * Partial Dependence plots, which are not currently supported for multiclass problems * Population Stability Index * Prediction Statistics Table * Binary classification specific plots: * ROC curve * Cumulative Lift and Gains Charts * Quantile Response Rate Plot * Actual vs Predicted Probabilities and Actual vs Predicted Log Odds .. _Scikit-learn: Scikit-learn ~~~~~~~~~~~~ The scikit-learn AutoDoc is designed to be very similar to the H2O-3 AutoDoc. In general, disparities between the AutoDocs are due to differences in H2O-3's and scikit-learn's algorithms. Here we highlight some notable differences: * **Feature Importance** is only provided for scikit-learn algorithms that support feature importance. * For **scikit-learn 0.22** and higher, AutoDoc shows permutation importance for models that don't have built-in feature importance. * **Validation Strategy** section is only included in the H2O-3 AutoDoc. * H2O-3 estimators include hold-out data and cross-validation information, while scikit-learn estimators do not. * **Prediction Probability Information** is limited to scikit-learn classifiers that provided prediction probabilities (accessible via the `predict_proba()` method). As a result, AutoDoc for classifiers without **predict_probab()** will exclude sections that require predicted probabilities: * Quantile Response Rates * Actual vs. Predicted, Prediction statistics * Population Stability Index. * Partial Dependence Plots See :ref:`scikit-autodoc-usage-ref` to know more. Dataset Dependent Sections --------------------------- The AutoDoc can render reports for models (Python Object, binary, MOJO, etc.,) with and without access to hold-out datasets. While H2O-3's AutoDoc can render model-only reports, **the scikit-learn AutoDoc requires that you pass in training data**. The following table summarizes AutoDocs with training data and/or hold-out datasets: +----------------------------+---------------+--------------------------+ | Section Topic | Model + Train | Model + Train + Hold-Out | +----------------------------+---------------+--------------------------+ | Shift Detection | X | | +----------------------------+---------------+--------------------------+ | Population Stability Index | X | | +----------------------------+---------------+--------------------------+ | Hold-Out Data Metrics | | X | +----------------------------+---------------+--------------------------+ If no datasets are provided, however, the report will exclude additional sections. The table below details the limitations of an H2O-3 model-only AutoDoc. +--------------------------+----------------+-------------+ | Section Name | Model and Data | Model | +--------------------------+----------------+-------------+ | Experiment Overview | X | X | +--------------------------+----------------+-------------+ | Data Overview | X | | +--------------------------+----------------+-------------+ | Validation Strategy | X | | +--------------------------+----------------+-------------+ | Feature Importance | X | Limited | +--------------------------+----------------+-------------+ | Final Model | X | Limited | +--------------------------+----------------+-------------+ | Alternative Models | X | X | +--------------------------+----------------+-------------+ | Partial Dependence Plots | X | | +--------------------------+----------------+-------------+ | Appendix | X | Limited | +--------------------------+----------------+-------------+ **Data Limitation Details by Section** Additional limitations can occur depending on the machine learning software platform (i.e., H2O-3, scikit-learn) and the software platform's version number. See the corresponding software sections for further details. * **Feature Importance**: * Excludes: * Shapley Feature Importance * **Final Model Section**: * Excludes: * Population Stability Index * Prediction Statistics * Quantile Response Plots * Actual vs. Predicted Probabilities and Actual vs. Predicted Log Odds * Can Include: * Confusion Matrix * H2O-3 Performance Charts: Scoring History, ROC Curve, Cumulative Lift and Cumulative Gains * **Appendix**: * Excludes: * Quantile Plot Calculation Tables (These tables only appear if quantile based table and plots are created) Supported File Types --------------------- H2O AutoDoc can generate the following file types: * Microsoft Word documents (.docx) * Markdown files (.md) Both of these file types can easily be exported to HTML or PDF. Change Log ---------- Version 1.0.5 (Feb 28, 2021) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Support for H2O-3 version 3.34.0.3 - Ability to push generated AutoDoc to S3 Bucket and GitHub Repository - Bug fixes and improvements on templates and documentation Version `1.0.4 `_ (Sep 17, 2021) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Support for Python 3.8 - Bug fixes and improvements on templates and documentation Version `1.0.3 `_ (Nov 11, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Documentation updates Version `1.0.2 `_ (August 27, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - New Features: - Allow Additional Testsets - Support Scikit-Learn Supervised Learning Models - Add AUCPR Plots - Add Cross-Validation Plots - Add Per Fold Cross-Validation Performance - Improvements: - Return full path to the AutoDoc - Show All Final Model Parameters - Bug Fixes: - Suppress Deprecation Warnings - Fix Key Error For Mojo with No Data - Fix Wrong "Generated On Date" for Mojo Models Version 1.0.1 (July 30, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Initial stable release