Skip to main content
Version: Next

Settings: Backtesting

Overview​

H2O Model Validation offers an array of settings for a backtesting test. Below, each setting is described in turn.

Metrics​

Test name​

This setting defines the name of the validation test. By default, H2O Model Validation assigns a name to the test that you can rewrite.

Model​

This setting defines the model H2O Model Validation utilizes to run the size dependency test.

Model train dataset​

note

Model train dataset refers to one of the model's informational points, not a setting. This informational point refers to the model's training dataset that H2O Model Validation utilizes during the test to assess model performance over time. H2O Model Validation splits the training dataset by time to simulate a model refitting process while observing and recording the model's accuracy over time. H2O Model Validation utilizes the training data before a specified split date as the training dataset while utilizing training data after the specified split date as the test dataset.

Time column​

This column defines the time column in the primary dataset that H2O Model Validation utilizes to split the primary dataset (training dataset) during the backtesting test.

Split dates​

Options

  • Auto
    • This option enables H2O Model Validation to automatically define the split dates, which creates the backtesting experiments.
  • Custom
    • This option enables you to determine the split dates, which creates the backtesting experiments.

Number of splits​

Defines the number of dates (splits) H2O Model Validation utilizes for the dataset splitting and model refitting process during the validation test. H2O Model Validation sets the specified dates in the past while being equally apart from one another to generate appropriate dataset splits.

Model forecast horizon​

note

Model forecast horizon refers to one of the model's informational points.

Unit (Model forecast horizon)​

note

Unit (Model forecast horizon) refers to one of the model's informational points.

Test period length​

caution

The total of the specified Training period length and Test period length can not exceed the model's training dataset length because H2O Model Validation makes the splits on that dataset.

  • Acceptable values can range between 1 and the total test and training period length.

This setting defines the length of the test dataset for each backtesting split, starting from a split date.

  • If you utilize a test dataset to build the model, H2O Model Validation utilizes the length of the test dataset as the test period length
  • If you did not utilize a test dataset to build the model, H2O Model Validation utilizes the model’s forecast horizon as the test period length

Unit (Test period length)​

This setting defines the data time unit H2O Model Validation uses for each test dataset split (for the Test period length setting).

Training period length​

caution

The total of the specified Training period length and Test period length can not exceed the model's training dataset length because H2O Model Validation makes the splits on that dataset.

  • Acceptable values can range between 1 and the total test and training period length.

This setting defines the length of the training dataset for each backtesting split, ending at a split date. By default, H2O Model Validation specifies this setting as Expanding, which indicates that H2O Model Validation uses all the data up to a split date as the training dataset.

Unit (Training period length)​

This setting defines the data time unit H2O Model Validation uses for each training dataset split (for the Training period length setting).

Delete test models and datasets from the DAI instance after finish​

This setting determines if H2O Model Validation should delete the artifacts created on the Connection. In this case, artifacts refer to experiments and datasets generated during the backtesting validation test. By default, H2O Model Validation checks this setting (enables it), and accordingly, H2O Model Validation deletes all artifacts because they are no longer needed after the validation test is complete.

Feedback