Metrics: Backtesting
H2O Model Validation offers an array of metrics in the form of graphs, a chart, a heatmap, and tables to understand a Backtesting test. Below, each metric is described in turn.
Graph: Backtesting {metric}
The Backtesting {metric} graph displays the dynamics of the backtested models' scorer value through time. In this case, {metric} refers to the model's scorer. In addition, you can use the graph to see the dynamics of the model accuracy while discovering if accuracy depends on time. You can also use the graph to investigate past environmental changes during data collection that led to drops in model performance.
- Cross-validation: Cross-Validation metric values calcualted on train datasets
- Back-test: Backtesting metric values calculated on test datasets
- Y-axis: Model's scorer (Backtesting
{metric}) - X-axis: Date (Backtesting splits)
Graph: Target distributions
The Target distributions graph displays the target distribution of the train and test dataset (datasets obtained from the Primary Dataset you defined). You can use this graph to investigate model accuracy drops in the past due to a change in the target variable through time.
- Train: Target distribution values from the training dataset
- Back-test: Target distribution values from the test dataset
Heatmap: Feature importance for different split dates
The Feature importance for different split dates heatmap visualizes the most important features of the backtested models. The heatmap is helpful when investigating how variable importance evolves.
- Rows: Raw input variables
- Columns: Backtesting splits
- Heatmap values: Feature importance scorers
- Submit and view feedback for this page
- Send feedback about H2O Model Validation to cloud-feedback@h2o.ai