Skip to main content
Version: v0.17.0

Release notes

v0.17.0 | November 10, 2023​

Overview​

To validate your machine learning models, the major update to H2O Model Validation v0.17.0 is as follows:

  • H2O Model Validation now supports a robustness test. H2O Model Validation's robustness test is designed to assess the robustness of a machine learning model against noisy datasets. The test perturbs the dataset with various methods to introduce noise based on the type of each column in the dataset.

This version also includes improvements to backtesting graphs for regression models, new features for drift detection and segment performance tests, the ability to view pairwise correlations between numerical columns in datasets, and support for Connections to H2O Driverless AI version 1.10.6.2 or less and AI Engines v0.5.5 or less. Other notable changes include:

  • The removal of the Worker feature.
  • The deprecation of h2oGPT-GM LLM endpoints.

These changes aim to provide you with a comprehensive toolkit to evaluate, analyze, and enhance the performance and reliability of your machine learning models and datasets.

Tests​

  • New: H2O Model Validation now supports a robustness test. H2O Model Validation's robustness test is designed to assess the robustness of a machine learning model against noisy datasets. The test perturbs the dataset with various methods to introduce noise based on the type of each column in the dataset. To learn more, see Robustness.
    • Why? Using H2O Model Validation's robustness test, you can assess your machine learning model's performance in real-world scenarios with noisy or corrupted data. The test helps identify potential issues and allows you to address them before deployment.
  • Backtesting
    • Improvement: For regression models, for a backtesting test, it's violin plots have been replaced with box plots (titled Target distributions). To learn more, see Target distributions (linear scale).
      • Why? The new graph enables you to clearly investigate model accuracy drops in the past due to a change in the target variable over time.
    • Improvement: For regression models, you can now change the target distribution graph from a linear to a log scale within the Target distributions graph for a backtesting test. To learn more, see Target distributions (linear scale).
      • Why? Logscale helps reading target distributions that are very skewed or have long tails.
    • Improvement: Now, in the feature importance heatmap for a backtesting test, the heatmap does not display features not utilized in the model. Also, the heatmap does not display the target variable. To learn more, see Feature importance for different split dates.
      • Why? It helps to clearly identify features utilized by the model.
  • Dirft detection
    • New: For a drift tection test, you can now view the following chart that displays all of a model's variables from top to bottom, where H2O Model Validation orders variables from highest to lowest population stability index (PSI) or drift score values: Features drift.
      • Why? Analyzing a feature with PSI and drift score helps in understanding the stability and reliability of the feature as a predictor, which is essential for making informed decisions based on the model's predictions. By identifying stable features, you can increase the accuracy and reliability of your model, and by identifying drifting features, you can take appropriate actions to address the issue and improve the model's performance.
  • Segment performance
    • New: Now, a segment performance adds the model's target variable to the test. To learn more, see Segment performance.
      • Why? Analyzing the worst and best dataset segments based on the model's target column can help identify problems within the data.
    • New: Now, when defining the settings for a segment performance test, you can select the metric to be applied to measure model performance. To learn more see, Metrics
      • Why? Analyzing the data of a model with the model's metric can enable you to clearly understand data problems within the context of the model's problem type.
  • Adversarial similarity
    • New: For an adversarial similarity test, the following new setting enables you to define the location (Connection) where H2O Model Validation should create the appropriate models to run the adversarial similarity test: Driverless AI instance to run the adversarial similarity models on.
    • Improvement: Now, the features partial dependency plot (PDP) for an adversarial similarity test only displays one single PDP line instead of one for each dataset. Also, the plot now displays the average probability for the concatenated dataset. To learn more, see Feature particial dependency plot (PDP).
  • Size dependency
    • Improvement: Now, a size dependency test can run with any applicable dataset. Before this change, one had to import a model with a linked test dataset. With this improvement, one can pick any dataset that follows the format of the model training dataset to start a size dependency test immediately.
      • Why? This improvement enables models without a test dataset to undergo a size dependency test.

Datasets​

  • New: Now, you can view a dataset's numeric correlations. In particular, H2O Model Validation now allows you to view pairwise correlations between numerical columns of a dataset. To learn more, see View a dataset's numeric correlations.
    • Why? Viewing pairwise correlations between numerical columns in H2O Model Validation helps identify the strength and direction of linear relationships between variables. Correlation analysis provides insights into how different variables are related, which can help select the most relevant features for modeling and identify potential data issues.

Models​

  • New: A model's summary now contains new components to display (for example, model structure). To learn more, see View a model's summary.
    • Why? Viewing a model summary within H2O Model Validation is important for several reasons. For example, by viewing the model summary, data scientists can gain insights into the strengths and weaknesses of the model, such as which features are most important for predicting the target variable, the performance of the model on different subsets of the data, and the complexity of the model.

Connections​

  • New: In the H2O Model Validation UI, you can now resume a paused Connection linked to an AI Engine. To learn more, see Resume a Connection.
    • Why? The ability to resume a Connection linked to an AI Engine through the H2O Model Validation UI eliminates the need to exit H2O Model Validation to My AI Engien Manager to restart the engine.
  • New: H2O Model Validation supports Connections to Driverless AI (DAI) version 1.10.6.2 or less. To learn more, see Create a Connection.
  • New: Now, H2O model Validation supports AI Engines v0.5.5 or less. To learn more, see Create a Connection.
  • Depecreated: H2O Model Validation removes the Worker feature. As a result, H2O Model Validation does not require you to define a Worker.

App settings​

  • New: You can utilize the llama.h2o.ai endpoint in case of an outage on gpt.h2o.ai. To learn more, see Define a large language model (LLM) source.
  • New: h2oGPT-GM LLM endpoints are no longer supported and therfore, not the default LLM endpoints. h2oGPT LLM endpoints are the default ones now.
    • Why? h2oGPT-GM endpoints have been deprecated, and h2oGPT endpoints are now stable.

Documentation​

  • New: All new features and settings for v0.17.0 have been documented.
Feedback