AutoDoc Custom Template Placeholders

The following sections describe placeholders for AutoDoc’s custom template feature.

Using placeholders

You can customize the content that appears in an AutoDoc report by using placeholders. When you insert a placeholder into a template, the content unique to that specific placeholder appears in the generated report in the location where you inserted it.

A placeholder is defined as follows:

{{p section.render('placeholder_name')}}

The following example shows how to define the Experiment Overview.DAI Experiment Pipeline Column Types placeholder:

{{p section.render('Experiment Overview.DAI Experiment Pipeline Column Types')}}

List of placeholders

The following is a list of available placeholders categories:

Experiment Overview

Placeholders related to the Experiment Overview:

Name

Description

Experiment Overview.DAI Experiment Pipeline Column Types

A table with different column types and type descriptions for DAI

Experiment Overview.DAI Experiment Pipeline Time Series

A table of the DAI time series settings and definitions for each setting

Experiment Overview.DAI GPU Specifications

A sentence indicating whether DAI used available GPUs

Experiment Overview.DAI Intro Model Goal

An introductory paragraph on the scorer the model is trying to optimize

Experiment Overview.DAI Iterative Tuning

A section describing the different iterative steps in the DAI experiment pipeline (that is, model, feature, target tuning, and feature evolution)

Experiment Overview.DAI Validation Schema Options

A documentation-type section that defines the different types of validation strategies available to the user

Experiment Overview.Performance

A summary performance table. This table includes the performance of each data split (e.g., train, validation, test) for the user-selected scorer

Data Overview

Placeholders related to Data Overview:

Name

Description

Data Overview.DAI NLP Feature Preprocessing

If applicable, includes information about the architecture of NLP-based transformations

Data Overview.DAI Target Transformation Description

A section describing whether target tuning was performed and what target transformations were tried

Data Overview.Intro

A table with the dataset information, such as file path with respect to where DAI is installed, number of rows and number of columns

Data Overview.template

This section points to the following placeholders: Data Overview.Intro, Data Overview.Training, Data Overview.Shift

Data Overview.Training

A table with the training data summary statistics. This placeholder is used in the standard AutoDoc. The content is similar to Data Overview.DAI Training Data Detailed but has less descriptive text and does not include information about missing values

Methodology

Placeholders related to Methodology:

Name

Description

Methodology.Assumptions

A high-level overview of DAI’s assumptions and limitations. This section includes details about whether a shift was detected between datasets

Methodology.DAI Assumptions Detailed

A section describing whether a user provided a validation dataset and whether a shift in distribution between datasets was detected. There is some overlap between this subsection and the Methodology.Assumptions section

Methodology.DAI Pipeline Formatted

This placeholder is similar to the Methodology.Pipeline template, but it has different formatting (for example, the headers use a different Word docx style)

Methodology.DAI Preventing Overfit

A documentation-type section that explains how DAI avoids overfitting in general

Methodology.Intro

A short sentence to introduce the Methodology section

Methodology.Settings

A section listing the experiment settings

Methodology.template

This placeholder calls the following placeholders: Methodology.Intro, Methodology.Assumptions, Methodology.Pipeline, Methodology.Settings

Data Sampling

Placeholders related to Data Sampling:

Name

Description

Data Sampling.DAI NLP Sampling

If applicable, this placeholder provides information about sampling done by NLP-based transformers

Validation Strategy

Placeholders related to Validation Strategy:

Name

Description

Validation Strategy.DAI Fold Information

This placeholder is designed to be used with the Validation Strategy.template - it is a placeholder of the Validation Strategy.template

Validation Strategy.DAI Recipe Validation

This template is designed to be used with the Validation Strategy.template - it is a placeholder of the Validation Strategy.template

Validation Strategy.DAI Time Series Validation

This template is designed to be used with the Validation Strategy.template - it is a placeholder of the Validation Strategy.template

Feature Evolution

Placeholders related to Feature Evolution:

Name

Description

Feature Evolution.DAI Iteration Plot

The iterative model plot from the DAI experiment page

Feature Evolution.DAI Top Feature Evolution Features

A table of the top features from the feature evolution phase

Feature Evolution.template

This section summarizes the feature evolution phase and some of the transformations that were tried

Feature Transformations

Placeholders related to Feature Transformations:

Name

Description

Feature Transformations.Feature Importance

The feature importance table

Feature Transformations.Intro

A section describing the feature importance this includes the feature importance table from the Feature Transformations.Feature Importance section

Feature Transformations.Permutation Feature Importance

A section describing the permutation feature importance. Note, permutation feature importance must be enabled in the AutoDoc expert settings for this section to render information

Feature Transformations.template

This template is used to call placeholders: Feature Transformation.Intro, Feature Transformations.Permutation Feature Importance, NLP.DAI NLP Detail

Final Model

Placeholders related to the Final Model:

Name

Description

Final Model.DAI All Feature Transformations

This placeholder is designed to go in an Appendix section. This section includes the full feature importance dataset and is not limited to a user-specified number of features

Final Model.DAI Final Model Architecture

A section describing the components of the final model

Final Model.DAI Final Model Components Table

A table with the final model’s summary components like the model type, model weight, number of folds, etc.

Final Model.DAI Final Model Performance Table

A table with the final model’s performance across available scorers

Final Model.DAI Final Model Performance Text

This template is meant to be called directly after the Experiment Overview.DAI Iterative Tuning placeholder. This placeholder includes a short paragraph about final model selection and a performance table

Final Model.DAI Model and Component Table

This section includes the model component table (i.e., this placeholder calls the Final Model.DAI Final Model Components Table), which shows information like the model type, model weight, number of folds, etc.

Final Model.DAI Model Objective Table

A table with the final models optimized loss function - this is different than the scorer a user selected. This placeholder is called by the Final Model.DAI Loss Function placeholder

Final Model.DAI Model Package Description

A table that provides the algorithm name, package name, version of the package and the packages primary documentation string. This placeholder is called by the Final Model.DAI Model Components placeholder

Final Model.DAI Models Evaluated Table

A table with the algorithms available in DAI and the reason an algorithm was or wasn’t selected for the final model. This placeholder is called by the Final Model.DAI Model Components placeholder

Final Model.Pipeline Overview

This placeholder is called by the Final Model.Pipeline placeholder and shows a table of the final model components. If there are base learners it shows the model components of the base learners

Final Model.template

This placeholder calls the following placeholders: Image Processing.template, Final Model.Pipeline, Final Model.Performance, Final Model.GLM, Final Model.PSI, Final Model.Prediction Statistics, Final Model.Response Rates, Final Model.Additional Performance

Generalized Linear Model (GLM)

Placeholders related to Generalized Linear Model (GLM):

Name

Description

GLM.template

If applicable, this section shows details about kLIME global coefficients and local reason codes

Literature

Placeholders related to Literature:

Name

Description

Literature.DAI Assumptions and Limitations Intro

A few sentences that introduce the assumptions and limitation sections that are specific to DAI modeling techniques

Literature.DAI Literature Intro

A sentence introducing the DAI Literature section

Literature.DAI Model References

A section providing a description and reference for models in DAI

Machine Learning Interpretability (MLI)

Placeholders related to Machine Learning Interpretability (MLI):

Name

Description

MLI.DAI MLI Intro

A section that includes documentation-style information about global interpretation techniques

MLI.DAI MLI Section

A section that contains tables of the kLIME top and bottom global coefficients, the kLIME plot, and the surrogate decision tree plot, as well as a documentation-style overview of local interpretations, followed by kLIME reason codes, ICE, and LOCO plot. Note the local interpretation based plots and table require that the user specifies individual records of interest with the Python client’s individual_rows parameter

MLI.KLIME Plot

A description of kLIME with the kLIME plot

MLI.KLIME Reason Code Text

A documentation-type section that describes kLIME reason codes

MLI.Local Interpretability Row Information

This placeholder is only available if the user-specified individual_rows are provided. This placeholder is called by the DAI MLI Section placeholder

MLI.Surrogate DT

The surrogate Decision Tree plot. Note this section requires an mli key or if running through the MLI UI make sure to run the surrogate decision tree explainer and the AutoDoc

Model Tuning

Placeholders related to Model Tuning:

Name

Description

Model Tuning.template

Similar to the Model Tuning.DAI Hyperparameters placeholder. This template is specific to the standard AutoDoc

Natural Language Processing (NLP)

Placeholders related to Natural Language Processing (NLP):

Name

Description

NLP.DAI NLP Detail

Similar to DAI NLP Assumption, but includes information about NLP transformer sampling and limitations and does not distinguish between image and NLP transformers (i.e., you will see NLP/Image in the body text of this sub template). This template is used in the standard AutoDoc

NLP.DAI NLP Pipeline Description

A documentation-type section provides an overview of DAI’s NLP transformation pipeline and defines the different stages

Partial Dependence Plots (PDP)

Placeholders related to Partial Dependence Plots (PDP):

Name

Description

Partial Dependence Plots.DAI Out-of-Range PDP

This sections shows out-of-range PDPs however, the Partial Dependence Plots.template can also show out of range values

Partial Dependence Plots.DAI Residual Analysis

A section describing residual analysis and providing partial dependence plots on the residuals

Partial Dependence Plots.DAI Sensitivity Analysis

Similar to the Partial Dependence Plot (PDP) template. This sub template includes additional explanations about sensitivity analysis in general and shows a records original feature values along with the ICE overlaid PDP. This template expects a user to pass in the individual_rows parameter to the Python client with records of interest

Partial Dependence Plots.template

A section describing how partial dependence plots work and showing the partial dependence plots. This section is used in the standard AutoDoc template

Appendix

Placeholders related to the Appendix:

Name

Description

Appendix.DAI Performance Metrics

A glossary of DAI performance metrics

Appendix.DAI References

A reference for the standard AutoDoc. The reference will appear for models that use a 《bagged ensemble》 (pasting)

Appendix.NLP Appendix

The full NLP architecture information.

Appendix.PSI_Appendix

The table used to calculate PSI

Appendix.Response_Rates_Appendix

The quantile-base plots calculation table.

Appendix.template

This template points to the Appendix.PSI, Appendix.Response_Rates_Appendix, and the Appendix.NLP Appendix. If the final model is or includes a GLM this section also include the full GLM coefficients tables and the documentation on how to understand the GLM coefficients table. If a user has set the AutoDoc to show all configurations, the full configuration table will be shown in the appendix. And lastly, if a bagged ensemble (pasting) is used, the reference for that model type is listed