Skip to main content

Task 3: Build model (experiment)

Now, let's develop a time series model capable of predicting the weekly sales (Weekly_Sales) of a department among 45 Walmart stores in different regions.

  1. In the H2O Driverless AI navigation menu, click DATASETS.
  2. In the Datasets table, click walmart_tts_small_train.csv.
  3. Select PREDICT.
  4. In the DISPLAY NAME box, enter:
    tutorial-2a
  5. Click TEST DATASET and select walmart_tts_small_test.csv.
  6. Click TARGET COLUMN and select Weekly_Sales.
  7. Click TIME COLUMN and select Date.
  8. Click TIME GROUPS COLUMNS and select Store and Dept (Date will be selected already).
    note

    Time group columns in a time series model are categorical variables that divide the data into distinct subsets based on time-related attributes. These columns enable the model to capture and account for variability due to different groups exhibiting unique patterns or behaviors over time. By defining time group columns, the model can better understand and predict the target variable by leveraging each group's unique characteristics. When it comes to predicting weekly sales, recognizing Store, Dept, and Date as group columns allow the model to capture the variability in sales patterns across different store-department-date combinations. This approach ensures the model can make more precise and robust predictions by leveraging the unique characteristics of each group.

  9. Click DONE.
  10. Click WEIGHT COLUMN and select sample_weight.
  11. Adjust the ACCURACY setting by clicking the +/- button until the accuracy reaches 1.
  12. Adjust the TIME setting by clicking the +/- button until the time reaches 5.
  13. Adjust the INTERPRETABILITY setting by clicking the +/- button until the interpretability reaches 10.
    note

    For this tutorial, let's try to generate a model that can be easily understood. Setting the interpretability to a minimum of >= 7 in an H2O Driverless AI experiment prompts the system to prioritize models that offer higher interpretability, even if they exhibit slightly lower accuracy. This preference stems from the fact that models with enhanced interpretability are easier to comprehend and explain, a crucial requirement in industries mandating model transparency. Moreover, increasing interpretability leads to the generation of simpler features by H2O Driverless AI. When interpretability reaches a certain threshold, the system produces a monotonically constrained model, enhancing transparency and interpretability further. This streamlined approach not only simplifies the understanding of metrics but also streamlines the elimination of potentially complex features that could pose challenges from an interpretability standpoint. It's essential to acknowledge the potential trade-off between interpretability and accuracy, underscoring the importance of carefully aligning the chosen interpretability level with the specific requirements of the use case.

  14. Click SCORER (RMSE) and select R2.
  15. Click LAUNCH EXPERIMENT.

Experiment setup page where you set experiment type, display name, and training, test, and validation sets. On this page are the time series settings, training settings, and access to the expert settings

note
  • (A) In the UNAVAILABLE COLUMNS AT PREDICTION TIME setting, you can specify columns that will not be available at prediction time. For this tutorial, we will assume all columns in the training dataset will be available. For those use cases in which certain columns will not be available, recognizing and adhering to what is "unavailable at prediction time" ensures that your time series model is trained and evaluated under realistic conditions, enhancing its reliability and generalization ability.
    • Suppose you access the UNAVAILABLE COLUMNS AT PREDICTION TIME setting. In that case, you can specify the following columns as "unavailable at prediction time": MarkDown1 to MarkDown5: These columns can be considered "unavailable columns at prediction time" if their values are not known or predictable for future time periods beyond the training data. Including these columns in the test dataset or using them for predictions would violate the principle of using only available information at the time of forecasting, potentially leading to unrealistic performance metrics and unreliable model predictions. The columns can be included if we know the future markdowns for sure and if the information will be available at the prediction time. In our case, we do have that information, and we will assume it will be available in the future.
    • In the UNAVAILABLE COLUMNS AT PREDICTION TIME setting, you can not select the following columns that are seen in the training and test dataset:
      • Target column (Weekly_Sales): This is the target column; by definition, it is not available at prediction time because it is what you are trying to predict.
      • Sample weight column (sample_weight): This column is derived from the IsHoliday column but specifically emphasizes holiday weeks with a weight. While IsHoliday is available at prediction time, sample_weight may be more specific to the training phase to handle the model training process rather than prediction. However, whether this is used during prediction depends on the model implementation. In our case, the sample_weight column is used as auxiliary data to aid the training process. During training, we want to emphasize a week containing a holiday because it can be the case that the company might want to prepare for times of high demand.
  • (B) The FORECAST HORIZON setting is set (detected) automatically to 26 weeks.
    • The test dataset covers 26 weeks, from 05/04/2012 to 10/26/2012 (inclusive). The count begins from the week ending on 05/04/2012, resulting in a total of 26 weeks. Therefore, the forecast horizon is also 26 weeks, matching the length of the test dataset.
      note

      In practice, you should set the forecast horizon based on the requirements of the forecasting task, and the test dataset should cover at least this horizon to evaluate the model's accuracy properly. However, the test dataset can be longer than the forecast horizon if you want to perform multiple rolling or expanding window evaluations. We will discuss this a bit further in the following section: Rolling-window-based predictions.

  • (C) The GAP BETWEEN TRAIN/TEST PERIOD setting is set (detected) automatically.
    • With the Dates being inclusive in the datasets, there's no time gap between the training and test datasets.
Feedback