Skip to main content
Version: v0.15.0

Compare validation tests

H2O Model Validation enables you to compare validation tests of the same type to discover insights.

Instructions​

To compare validation tests, consider the following instructions:

  1. In the H2O Model Validation navigation menu, click Tests.
  2. Click the Select tests toggle.
  3. In the tests table, select at least two validation tests of the same type.
    note

    You can select multiple validation tests of different types, but you must select at least two of the same type. H2O Model Validation compares validation tests of the same type (e.g., Backtesting); therefore, validation tests of different types are not compared. H2O Model Validation organizes comparisons into tabs, each containing validation tests of a certain type: comparison-tabs.png

  4. Click Compare.
    note

    When comparing validation tests, H2O Model Validation displays specific metrics based o the compared validation tests. To learn more, see:

Comparison metrics: Validation tests​

H2O Model Validation offers certain comparison metrics based on the compared validation tests. Metrics are displayed in graphs, histograms, charts, etc.

Adversarial similarity​

H2O Model Validation offers several metrics in graphs to understand the adversarial similarity tests you compare. Below, each metric is described in turn.

Graph: AUC score​

The AUC score graph displays the AUC score given to each adversarial similarity test.

adversarial-similarity-acu-score.png

Bar graph: Feature importance​

The Feature importance bar graph displays the gain of the features in the adversarial similarity tests. Gain refers to the relative contribution of a feature towards the predictive values. A higher gain value implies a higher impact on generating predictions.

adversarial-similarity-feature-importance.png

Backtesting​

H2O Model Validation offers several metrics in graphs and a histogram to understand the backtesting validation tests you compare. Below, each metric is described in turn.

Graph: Test results​

The Test results graph displays the Back-test values for each split date of the backtesting tests, where Back-test refers to the target distribution values from the test dataset.

backtesting-test-results.png

Graph: Validation results​

The Validation results graph displays the Cross-validation values for each split date for the backtesting models. This graph can be helpful when estimating a model's fitness level to a dataset not used when training the model.

backtesting-validation-results.png

Drift detection​

H2O Model Validation offers several metrics in the form of a bar graph to understand the drift detection tests you compare.

Bar graph: PSI scores​

The PSI scores bar graph displays the Population Stability Index (PSI) for the features in the drift detection tests.

drift-detection-psiScores.png

Size dependency​

H2O Model Validation offers several metrics in the form of graphs to understand the size dependency tests you compare. Below, each metric is described in turn.

Graph: Test results​

The Test results graph displays the test {metric} values for the size dependency tests obtained with different test data sizes. {Metric}, in this case, refers to the scorer of the model of a validation test (for example, Root Mean Square Error (RMSE)).

size-dependency-test-results.png

Graph: Validation results​

The Test results graph displays the validation {metric} values for the size dependency tests obtained with different validation data sizes. {Metric}, in this case, refers to the scorer of the model of a validation test (for example, Root Mean Square Error (RMSE)).

size-dependency-validation-results.png

Feedback