Skip to main content
Version: v0.17.0

Settings: Calibration score

Overview​

H2O Model Validation offers an array of settings for a calibration score test. Below, each setting is described in turn.

Settings​

Test name​

This setting defines the name for the validation test. By default, H2O Model Validation assigns a name to the test that you can rewrite.

Model​

This setting defines the model H2O Model Validation utilizes to run the calibration score test.

Model training dataset​

note

Model training dataset refers to one of the model's informational points, not a setting. This informational point refers to the model's training dataset that H2O Model Validation utilizes during the validation test to assess how well the probabilities estimated by a classification model align with the actual event frequencies.

Primary dataset​

This setting defines the dataset the test utilizes to assess the model's calibration. H2O Model Validation applies the model to the dataset and calculates the Brier score per target class. Right after, H2O Model Validation groups the data into several buckets based on the values of the estimated event probabilities to calculate average probabilities and realized event frequencies.

note

The defined dataset (primary dataset) must follow the model's training dataset format.

Number of bins​

This setting defines the number of bins H2O Model Validation utilizes to divide the primary dataset.

info

This setting influences the calibration score graph. To learn more, see Calibration curve.

Binning strategy​

This setting defines the binning strategy H2O Model Validation utilizes to bin the primary dataset.

Options

  • Quantile
    • A Quantile binning strategy groups the records to have an equal number of records per bin.
  • Uniform
    • A Uniform binning strategy creates bins with equally sized ranges of probabilities.

Feedback