Skip to main content
Version: v1.3.0

Tutorial 1A: Default hyperparameter values (start your first experiment)

Overview​

This tutorial builds an image regression model capable of predicting the sum of Brazilian Real (R$) coins in images. While training this model, we will see how H2O Hydrogen Torch enables you to generate accurate models with expertly default hyperparameter values that H2O Hydrogen Torch derives from model training best practices used by top Kaggle grandmasters.

Also, we will discover how H2O Hydrogen Torch enables you to monitor and understand the impact of selected hyperparameter values on the training process through simple interactive charts.

Prerequisites​

Step 1: Import dataset​

For this tutorial, we use the preprocessed Coins image regression dataset. The dataset contains a collection of 6,028 images with one or more coins. Each image has been labeled to indicate the sum of its coins. The currency of the coins is the Brazilian Real (R$). Let's import the dataset:

  1. In the H2O Hydrogen Torch navigation menu, click Import dataset.
  2. In the S3 file name list, select coins_image_regression.zip.
  3. Click Continue.
  4. Again, click Continue.
  5. Again, click Continue.
Note

H2O Hydrogen Torch requires the dataset for an experiment to be preprocessed to follow a certain dataset format for the problem type the experiment aims to solve. The Coins image regression dataset was preprocessed to follow a dataset format for an image regression model. To learn more, see Dataset formats.

Step 2: Explore dataset​

Let's explore the Coins image regression dataset.

  1. In the H2O Hydrogen Torch navigation menu, click View datasets.
  2. In the datasets table, click coins_image_regression.

On the Sample train data tab, we can observe 3 columns and few of the 6,028 rows in the Coins image regression dataset. Each row in the dataset contains the name of an image (image_path) depicting several coins that add up to a certain sum recorded in the label column.

Sample train data tab for coins_image_regression

To better understand the dataset, let's view a few visual examples of the dataset rows.

  1. Click the Sample train visualization tab.

    If you hover over one of the images, the label (sum) of the image appears.

    Sample train visulization tab for coins_image_regression

Now that we better understand the Coins image regression dataset let's build a model that predicts the sum of coins an image contains.

Step 3: Build model (experiment)​

Using default values for hyperparameters, let's build our image regression model.

  1. In the H2O Hydrogen Torch navigation menu, click Create experiment.
  2. In the Dataset list, select coins_image_regression. After selecting the coins_image_regression dataset, H2O Hydrogen Torch displays settings for an image regression experiment. H2O Hydrogen Torch autodetects the dataset type and displays appropriate settings. For all the settings available for an image regression model, H2O Hydrogen Torch has defined each setting with a default value while considering model training best practices used by top Kaggle grandmasters. To learn about the available settings for an image regression experiment, see Experiment settings: Image regression.
  3. In the Experiment name box, enter tutorial-1a.
  4. Click Run experiment.
    Note
    • A model is evaluated using a model scorer, and for our model, the default scorer is the Mean Absolute Error (MAE). For our model purposes, note the following points:
      • We want a validation score of 0, indicating an excellent prediction value where all predictions' loss is 0
      • Loss refers to the penalty for a bad prediction. Loss indicates how bad the model's prediction was on a single example. A loss value of zero for a given single example indicates that the model's prediction is perfect
      • The closer the MAE value (validation score) approximates to 0, the better (in regards to each prediction having a lower loss value). The validation score represents, on average, how much the model predictions (sums) are off compared to the true sums. Accordingly, the model being off also refers to loss, which indicates the loss is greater than zero
    • After starting your experiment, H2O Hydrogen Torch takes you to the View experiments card, where you can view running and completed experiments.

Step 4: Observe running experiment (model)​

As the experiment completes, let's observe the performance of our image regression model through simple interactive charts.

Caution

The prediction metrics from your experiment might differ from those discussed from this point on.

  1. In the experiments table, click tutorial-1a

    note

    As the experiment completes, refresh the charts. To refresh the charts:

    1. Click Refresh.

    Hovering over the Validation MAE chart, we see that after the second epoch is completed, the MAE decreases to 3.957. Recall, the closer the MAE value (validation score) approximates to 0, the better.

    Validation MAE chart with a step value of 14400, and a MAE value of 3.957

    To visualize the impact of this MAE decrease, let's see what the Validation prediction insights tab contains.

  2. Click the Validation prediction insights tab.

    In this case, in the Random validation samples tab, we can see random validation samples highlighting the best, and worst validation samples as each Epoch is complete.

    Random validation samples tab with three validation samples rated from best to worst

    Viewing several validation samples can tell us how the experiment is doing in terms of achieving a low (good) validation score. Hovering on one of the random validation samples, we can see the predicted and actual sum of the image. As well, each validation sample contains a Loss value.

Step 5: Observe completed experiment (model)​

After a few minutes, you can observe the completed experiment (model). Let's observe our completed experiment.

  1. In the H2O Hydrogen Torch navigation menu, click View experiments.
  2. In the experiments table, locate experiment tutorial-1a. tutorial-1a listed in the experiments table with a finished status
    • Observe the experiment's final validation metric in the val metric column (2.653).
    • H2O Hydrogen Torch marks an experiment as completed when its status changes to finished.

To further understand the final validation score, we can reexplore the Validation prediction insights tab to learn from all the best and worst validation samples now that the experiment is complete. In particular, we can explore the type of images the model was good or bad at predicting.

  1. In the experiments table, click tutorial-1a.

  2. Click the Validation prediction insights tab.

  3. Click the Best validation samples tab.

    Here, we can see the best validation samples where a prediction's loss value is low when compared to other samples.

    Validation prediction samples tab with Best validation samples tutorial-1a

  4. Click the Worst validation samples tab.

    Here, we can see the worst validation samples where a prediction's loss value is high when compared to other samples.

    Validation prediction samples tab with Worst validation samples tutorial-1a

Observing the best and worst validation samples shows that high loss values were generated anytime the image was too dark. With this in mind, we can assume that replacing these dark images with more clear ones can lead to a better validation score because clear images allow our regression model to learn from clear images where coins can be distinguished.

At this point, we might be wondering when it will be good or bad to implement this built model in production. Usually, a well-performing model is defined by an acceptable solution within the constraints and expectations of the use case. For example:

  • Use case 1: Suppose an Automated Teller Machine (ATM) accepts coins as a deposit while using our built model. In this scenario, our built model's validation MAE can be viewed as low-performing, leading to someone's coins deposit being over or undercounted. Not suitable for the bank ATM if the model overcounts and good for the customer if it does. The model is not acceptable under this use case because an ATM is expected to count accurately. With that in mind, we need to tune the hyperparameters to improve the validation MAE.
  • Use case 2: Suppose you want to obtain a rapid approximate sum of many coins using our built model. In this scenario, we first need to define an acceptable margin of error for a rapid approximation. If our built model's validation MAE can match the defined acceptable margin of error, we can say the model in this scenario is acceptable (well-performing).

Summary​

In this tutorial, we learned that H2O Hydrogen Torch enables you to generate accurate models with expertly default hyperparameter values that H2O Hydrogen Torch derives from model training best practices used by top Kaggle grandmasters. Also, we discovered that H2O Hydrogen Torch provides several interactive graphs to monitor and understand performance metrics and predictions during and after an experiment.

Next​

You can improve the built image regression model; in particular, you can use grid search (a feature of H2O Hydrogen Torch) to improve the model. To learn more, consider the following tutorial:

Once you have trained or improved your model, you can deploy the model into production. H2O Hydrogen Torch supports three ways to deploy a built model. You can deploy a model using the H2O Hydrogen Torch UI, a model's Python scoring pipeline, and a model's H2O MLOps pipeline.

Feedback