Skip to main content
Version: v0.16.0

Settings: Calibration score

H2O Model Validation offers an array of settings for a calibration score test. Below, each setting is described in turn.

Test name​

Defines the name for the validation test; by default, H2O Model Validation assigns a name to the test that you can rewrite.

Model​

Defines the model H2O Model Validation utilizes to run the calibration score test.

Model training dataset​

note

Model training dataset refers to one of the model's informational points, not a setting. This informational point refers to the model's training dataset that H2O Model Validation utilizes during the validation test to assess how well the probabilities estimated by a classification model align with the actual event frequencies.

Primary dataset​

Defines the dataset the test utilizes to assess the model's calibration. H2O Model Validation applies the model to the dataset and calculates the Brier score per target class. Right after, H2O Model Validation groups the data into several buckets based on the values of the estimated event probabilities to calculate average probabilities and realized event frequencies.

The defined dataset (primary dataset) must follow the model's training dataset format.

Number of bins​

Defines the number of bins H2O Model Validation utilizes to divide the primary dataset.

info

This setting influences the calibration score graph. To learn more, see Graph: Calibration score.

Binning strategy​

Defines the binning strategy H2O Model Validation utilizes to bin the primary dataset.

Options

  • Quantile
    • A Quantile binning strategy groups the records to have an equal number of records per bin.
  • Uniform
    • A Uniform binning strategy creates bins with equally sized ranges of probabilities.

Feedback