AutoDoc Custom Template Placeholders¶

The following sections describe placeholders for AutoDoc’s custom template feature.

Using placeholders¶

You can customize the content that appears in an AutoDoc report by using placeholders. When you insert a placeholder into a template, the content unique to that specific placeholder appears in the generated report in the location where you inserted it.

A placeholder is defined as follows:

{{p section.render('placeholder_name')}}

The following example shows how to define the Experiment Overview.DAI Experiment Pipeline Column Types placeholder:

{{p section.render('Experiment Overview.DAI Experiment Pipeline Column Types')}}

List of placeholders¶

The following is a list of available placeholders categories:

Experiment Overview
Data Overview
Methodology
Data Sampling
Validation Strategy
Feature Evolution
Feature Transformations
Final Model
Generalized Linear Model (GLM)
Literature
Machine Learning Interpretability (MLI)
Model Tuning
Natural Language Processing (NLP)
Partial Dependence Plots (PDP)
Appendix

Experiment Overview¶

Placeholders related to the Experiment Overview:

Name	Description
Experiment Overview.DAI Experiment Pipeline Column Types	A table with different column types and type descriptions for DAI
Experiment Overview.DAI Experiment Pipeline Time Series	A table of the DAI time series settings and definitions for each setting
Experiment Overview.DAI GPU Specifications	A sentence indicating whether DAI used available GPUs
Experiment Overview.DAI Intro Model Goal	An introductory paragraph on the scorer the model is trying to optimize
Experiment Overview.DAI Iterative Tuning	A section describing the different iterative steps in the DAI experiment pipeline (that is, model, feature, target tuning, and feature evolution)
Experiment Overview.DAI Validation Schema Options	A documentation-type section that defines the different types of validation strategies available to the user
Experiment Overview.Performance	A summary performance table. This table includes the performance of each data split (e.g., train, validation, test) for the user-selected scorer

Data Overview¶

Placeholders related to Data Overview:

Name	Description
Data Overview.DAI NLP Feature Preprocessing	If applicable, includes information about the architecture of NLP-based transformations
Data Overview.DAI Target Transformation Description	A section describing whether target tuning was performed and what target transformations were tried
Data Overview.Intro	A table with the dataset information, such as file path with respect to where DAI is installed, number of rows and number of columns
Data Overview.template	This section points to the following placeholders: Data Overview.Intro, Data Overview.Training, Data Overview.Shift
Data Overview.Training	A table with the training data summary statistics. This placeholder is used in the standard AutoDoc. The content is similar to Data Overview.DAI Training Data Detailed but has less descriptive text and does not include information about missing values

Methodology¶

Placeholders related to Methodology:

Name	Description
Methodology.Assumptions	A high-level overview of DAI’s assumptions and limitations. This section includes details about whether a shift was detected between datasets
Methodology.DAI Assumptions Detailed	A section describing whether a user provided a validation dataset and whether a shift in distribution between datasets was detected. There is some overlap between this subsection and the Methodology.Assumptions section
Methodology.DAI Pipeline Formatted	This placeholder is similar to the Methodology.Pipeline template, but it has different formatting (for example, the headers use a different Word docx style)
Methodology.DAI Preventing Overfit	A documentation-type section that explains how DAI avoids overfitting in general
Methodology.Intro	A short sentence to introduce the Methodology section
Methodology.Settings	A section listing the experiment settings
Methodology.template	This placeholder calls the following placeholders: Methodology.Intro, Methodology.Assumptions, Methodology.Pipeline, Methodology.Settings

Data Sampling¶

Placeholders related to Data Sampling:

Name	Description
Data Sampling.DAI NLP Sampling	If applicable, this placeholder provides information about sampling done by NLP-based transformers

Validation Strategy¶

Placeholders related to Validation Strategy:

Name	Description
Validation Strategy.DAI Fold Information	This placeholder is designed to be used with the Validation Strategy.template - it is a placeholder of the Validation Strategy.template
Validation Strategy.DAI Recipe Validation	This template is designed to be used with the Validation Strategy.template - it is a placeholder of the Validation Strategy.template
Validation Strategy.DAI Time Series Validation	This template is designed to be used with the Validation Strategy.template - it is a placeholder of the Validation Strategy.template

Feature Evolution¶

Placeholders related to Feature Evolution:

Name	Description
Feature Evolution.DAI Iteration Plot	The iterative model plot from the DAI experiment page
Feature Evolution.DAI Top Feature Evolution Features	A table of the top features from the feature evolution phase
Feature Evolution.template	This section summarizes the feature evolution phase and some of the transformations that were tried

Feature Transformations¶

Placeholders related to Feature Transformations:

Name	Description
Feature Transformations.Feature Importance	The feature importance table
Feature Transformations.Intro	A section describing the feature importance this includes the feature importance table from the Feature Transformations.Feature Importance section
Feature Transformations.Permutation Feature Importance	A section describing the permutation feature importance. Note, permutation feature importance must be enabled in the AutoDoc expert settings for this section to render information
Feature Transformations.template	This template is used to call placeholders: Feature Transformation.Intro, Feature Transformations.Permutation Feature Importance, NLP.DAI NLP Detail

Final Model¶

Placeholders related to the Final Model:

Name	Description
Final Model.DAI All Feature Transformations	This placeholder is designed to go in an Appendix section. This section includes the full feature importance dataset and is not limited to a user-specified number of features
Final Model.DAI Final Model Architecture	A section describing the components of the final model
Final Model.DAI Final Model Components Table	A table with the final model’s summary components like the model type, model weight, number of folds, etc.
Final Model.DAI Final Model Performance Table	A table with the final model’s performance across available scorers
Final Model.DAI Final Model Performance Text	This template is meant to be called directly after the Experiment Overview.DAI Iterative Tuning placeholder. This placeholder includes a short paragraph about final model selection and a performance table
Final Model.DAI Model and Component Table	This section includes the model component table (i.e., this placeholder calls the Final Model.DAI Final Model Components Table), which shows information like the model type, model weight, number of folds, etc.
Final Model.DAI Model Objective Table	A table with the final models optimized loss function - this is different than the scorer a user selected. This placeholder is called by the Final Model.DAI Loss Function placeholder
Final Model.DAI Model Package Description	A table that provides the algorithm name, package name, version of the package and the packages primary documentation string. This placeholder is called by the Final Model.DAI Model Components placeholder
Final Model.DAI Models Evaluated Table	A table with the algorithms available in DAI and the reason an algorithm was or wasn’t selected for the final model. This placeholder is called by the Final Model.DAI Model Components placeholder
Final Model.Pipeline Overview	This placeholder is called by the Final Model.Pipeline placeholder and shows a table of the final model components. If there are base learners it shows the model components of the base learners
Final Model.template	This placeholder calls the following placeholders: Image Processing.template, Final Model.Pipeline, Final Model.Performance, Final Model.GLM, Final Model.PSI, Final Model.Prediction Statistics, Final Model.Response Rates, Final Model.Additional Performance

Generalized Linear Model (GLM)¶

Placeholders related to Generalized Linear Model (GLM):

Name	Description
GLM.template	If applicable, this section shows details about kLIME global coefficients and local reason codes

Literature¶

Placeholders related to Literature:

Name	Description
Literature.DAI Assumptions and Limitations Intro	A few sentences that introduce the assumptions and limitation sections that are specific to DAI modeling techniques
Literature.DAI Literature Intro	A sentence introducing the DAI Literature section
Literature.DAI Model References	A section providing a description and reference for models in DAI

Machine Learning Interpretability (MLI)¶

Placeholders related to Machine Learning Interpretability (MLI):

Name	Description
MLI.DAI MLI Intro	A section that includes documentation-style information about global interpretation techniques
MLI.DAI MLI Section	A section that contains tables of the kLIME top and bottom global coefficients, the kLIME plot, and the surrogate decision tree plot, as well as a documentation-style overview of local interpretations, followed by kLIME reason codes, ICE, and LOCO plot. Note the local interpretation based plots and table require that the user specifies individual records of interest with the Python client’s `individual_rows` parameter
MLI.KLIME Plot	A description of kLIME with the kLIME plot
MLI.KLIME Reason Code Text	A documentation-type section that describes kLIME reason codes
MLI.Local Interpretability Row Information	This placeholder is only available if the user-specified individual_rows are provided. This placeholder is called by the DAI MLI Section placeholder
MLI.Surrogate DT	The surrogate Decision Tree plot. Note this section requires an mli key or if running through the MLI UI make sure to run the surrogate decision tree explainer and the AutoDoc

Model Tuning¶

Placeholders related to Model Tuning:

Name	Description
Model Tuning.template	Similar to the Model Tuning.DAI Hyperparameters placeholder. This template is specific to the standard AutoDoc

Natural Language Processing (NLP)¶

Placeholders related to Natural Language Processing (NLP):

Name	Description
NLP.DAI NLP Detail	Similar to DAI NLP Assumption, but includes information about NLP transformer sampling and limitations and does not distinguish between image and NLP transformers (i.e., you will see NLP/Image in the body text of this sub template). This template is used in the standard AutoDoc
NLP.DAI NLP Pipeline Description	A documentation-type section provides an overview of DAI’s NLP transformation pipeline and defines the different stages

Partial Dependence Plots (PDP)¶

Placeholders related to Partial Dependence Plots (PDP):

Name	Description
Partial Dependence Plots.DAI Out-of-Range PDP	This sections shows out-of-range PDPs however, the Partial Dependence Plots.template can also show out of range values
Partial Dependence Plots.DAI Residual Analysis	A section describing residual analysis and providing partial dependence plots on the residuals
Partial Dependence Plots.DAI Sensitivity Analysis	Similar to the Partial Dependence Plot (PDP) template. This sub template includes additional explanations about sensitivity analysis in general and shows a records original feature values along with the ICE overlaid PDP. This template expects a user to pass in the `individual_rows` parameter to the Python client with records of interest
Partial Dependence Plots.template	A section describing how partial dependence plots work and showing the partial dependence plots. This section is used in the standard AutoDoc template

Appendix¶

Placeholders related to the Appendix:

Name	Description
Appendix.DAI Performance Metrics	A glossary of DAI performance metrics
Appendix.DAI References	A reference for the standard AutoDoc. The reference will appear for models that use a “bagged ensemble” (pasting)
Appendix.NLP Appendix	The full NLP architecture information.
Appendix.PSI_Appendix	The table used to calculate PSI
Appendix.Response_Rates_Appendix	The quantile-base plots calculation table.
Appendix.template	This template points to the Appendix.PSI, Appendix.Response_Rates_Appendix, and the Appendix.NLP Appendix. If the final model is or includes a GLM this section also include the full GLM coefficients tables and the documentation on how to understand the GLM coefficients table. If a user has set the AutoDoc to show all configurations, the full configuration table will be shown in the appendix. And lastly, if a bagged ensemble (pasting) is used, the reference for that model type is listed