Skip to main content

H2O AI Hybrid Cloud release notes

v23.10.2 | April 05, 2024​

Important

End of support date: Jan 23, 2025.

For more information, see About version support.

Upgraded components​

The following components were upgraded in this minor release:

  • H2O MLOps v0.62.5.

New features​

H2O MLOps​

  • Added support for enabling health monitoring without drift. You can now enable Health monitoring (prediction volume and scoring latency) without Drift monitoring (which is dependent on raw data). This means that there are three total possible monitoring settings for a given deployment in H2O MLOps:
    • monitoring disabled
    • monitoring enabled for health only (no raw input data stored)
    • monitoring enabled for health and drift (raw input data stored)
      info

      H2O MLOps deployments will start with model monitoring disabled by default.

  • Added the ability to disable storing score transaction data in monitoring via the Deployer API

H2O Enterprise Steam​

  • Fixed a bug where the platform host name was not propagated to Discovery annotations.

AI App Store​

  • Fixed a bug with the custom image regex that was set for AI Appstore.

Known issues and workarounds​

Known IssueWorkaround
Timeout during helm install for H2O Notebook application when using the GKE filestore, due to PVC creation delay.Add timeout = 600 to line 66 in terraform/modules/applications/notebook/notebook.tf.

v23.10.1 | March 15, 2024​

Important

End of support date: Jan 23, 2025.

For more information, see About version support.

Upgraded components​

The following components were upgraded in this minor release:

Core components​

  • H2O MLOps v0.62.4

Optional components​

  • H2O Document AI v0.7.2

New features​

Platform installer​

  • H2O Document AI and H2O MLAPI are now enabled separately on the installer. To enable both H2O Document AI and H2O MLAPI, set optional_applications_to_install = ["docai", "mlapi"].

H2O MLOps​

Frontend

  • Mitigating against potential XSS attacks.
    • Content Security Headers (CSP) are set on the frontend to minimise potential XSS attacks fetched through any untrusted sources
    • Sanitised all plain text inputs (name, description) to minimise potential XSS attacks.
    • Validated other text-based fields to avoid accepting XSS vulnerable characters (. < > " ') as the user input.
    • Sanitised all text-based downstream response fields (name, description) to minimise potential XSS attacks.
  • Introduced access control based on JWT claims. This means you can now control access to the H2O MLOps frontend UI based on user role.

Storage

  • The CreateArtifact API and UploadArtifact API now validates against a known list of mime types to allow or reject an artifact creation/upload respectively. The default configuration is to allow all mime types.
  • All user text inputs are escaped and stored within the database to avoid potential XSS issues.

Deployer

  • All user text inputs are escaped and stored within the component to avoid potential XSS issues.
info

For a full list of features, improvements, and bug fixes in H2O MLOps, see the H2O MLOps release notes.

H2O Document AI​

  • Added a new proxy version and updated the helm chart to reflect the new proxy URL.
note
  • If LinkerD is enabled, restart your H2O DocAI scorers after the upgrade. Otherwise the scorers may not work.
  • To enable LinkerD for the H2O DocAI viewer, add the LinkerD annotation (linkerd.io/inject: enabled) annotations on the api.podAnnotations and ui.podAnnotations sections. Do not annotate the document-ai-viewer namespace with the LinkerD annotation as archiveExtract jobs do not support the LinkerD injection yet.

Known issues and workarounds​

Known IssueWorkaround
Timeout during helm install of the H2O Notebook application when using the GKE filestore, due to PVC creation delay.Add timeout = 600 to line 66 in terraform/modules/applications/notebook/notebook.tf.
MLOps model upload fails after TF migration and fresh installs due to a UID mismatch on the pod and PVC.Add the following code snippet to line 609 in terraform/modules/applications/mlops-helm/resources/values.yaml. 

containerSecurityContext:
  # -- Security context for the main containers.
  enabled: true
  runAsNonRoot: true
  readOnlyRootFilesystem: false
  allowPrivilegeEscalation: false
  seccompProfile:
    type: "RuntimeDefault"
  capabilities:
    drop:
      - ALL
podSecurityContext:
  # -- Whether a pod security context is applied to each pod 
deployed in the environment.
  enabled: true
  runAsNonRoot: true
  runAsUser: 1001
  runAsGroup: 1001
  fsGroup: 1001

v23.10.0 | Jan 23, 2024​

Important

End of support date: Jan 23, 2025.

For more information, see About version support.

New components​

The following components are optional components of new installations of HAIC.

  • AI Notebooks v0.1.10
  • H2O eScorer v0.7.0
  • Enterprise h2oGPTe
note

Let your H2O contact know if you would like to install these applications into your AI App Store. Enterprise h2OGPTe is not yet a part of H2O AI Cloud, however, it can be installed within the same cluster.

Upgraded components​

Core components​

  • H2O Driverless AI v1.10.6.2
  • H2O MLOps v0.62.2
  • Enterprise Steam v1.9.7
  • AI App Store v0.29.1
  • H2O-3 v3.44.0.2
  • H2O Discovery Service v1.2.6
  • H2O Logging Service v0.1.4

Optional components​

Optional components of HAIC are provided to customers as needed based on your specific requirements. If you have any of the following components, note the following upgrades.

  • H2O Hydrogen Torch v1.3.3
  • H2O Document AI v0.7.0
  • H2O Model Validation v0.17.0
  • H2O Feature Store v0.19.3
  • H2O AI Engine Manager v0.5.5
  • H2O Autodoc v0.9.0
  • H2O Label Genie v0.4.1

New features​

Platform​

  • Added Private CA certificate support for H2O AI Hybrid Cloud runtimes. This means that H2O AI Hybrid Cloud now supports using certificates generated by a private certificate authority.
  • Supports backup and restore using Velero.
  • H2O AI Hybrid cloud now adopts a standardized image naming scheme for H2O MLOps DB servers that are deployed outside of the Kubernetes cluster. For example,
    • Old naming convention: <image-registry>/artifact-fetcher:0.62.2
    • New naming convention: <image-registry>/h2oai-modeldeployment-artifactfetcher:v0.62.2
  • Upgraded Postgres to v14.8
  • Compatibility up to Kubernetes v1.28
  • Deprecated the following AI App Store runtimes:
    • ub1804_cuda110_cudnn8_py37_wlatest: Python 3.7 GPU Runtime
    • deb10_py37_wlatest: Python 3.7 CPU Runtime.
  • The new default AI App Store runtime is deb11_py310_wlatest (Python 3.10 CPU Runtime).

H2O Driverless AI​

  • Added integration with h2oGPT that lets you optionally generate and view dataset or experiment summaries using a GPT model. For more information, see h2oGPT integration.
  • Added support for Okta SSO authenticator on the Snowflake connector.
  • Added support for custom telemetry recipes.
  • Added support for a PyTorch backend in Triton server.
  • Added support for sharing image datasets to H2O Storage. For more information, see H2O Storage (remote storage) integration.
  • You can now tag experiments inside Projects that are connected to H2O Storage (remote storage). These tags are also displayed in H2O MLOps. For more information, see Experiment tagging.
  • DAI Model Dashboard: You can now view a new DAI model dashboard that provides comprehensive insights into the performance of models created using DAI. For more information, see MLI Dashboard.
  • Performance Charts: You can now view performance charts specifically designed for Decision Tree (DT) and Random Forest (RF) surrogate models.
  • NLP LOCO Scoring Pipeline: Added the ability to construct a NLP LOCO scoring pipeline.
  • The Python client now supports the generation of plots for individual explainers. Supported explainers include:
    • Partial Dependence Plot
    • Shapley Summary Plot for Original Features (Naive Shapley Method)
    • Shapley Values for Transformed Features
    • Shapley Values for Original Features (Naive Method)
    • Surrogate Decision Tree
  • Added new configuations for the Google BigQuery (GBQ) data connector.
info

For a full list of features, improvements, and bug fixes in H2O Driverless AI, see the H2O Driverless AI v1.10.5.1, v1.10.6, v1.10.6.1, & v1.10.6.2 release notes.

H2O MLOps​

  • For GPU-enabled model deployments, you can now set the appropriate Kubernetes (K8s) requests and limits by clicking the GPU Deployment toggle when creating a deployment. For more information, see Deploy a model and Kubernetes options.
  • You can now create and assign experiment tags within a project. For more information, see Project page tabs and Add experiments.
  • You can now edit the names and tags of experiments. For more information, see Project page tabs.
  • Added support for up to H2O Driverless AI v1.10.6.1 runtime
  • The ListExperiments API can now be used to filter experiments by status (ACTIVE, DELETED). By default, the API returns ACTIVE experiments.
info

For a full list of features, improvements, and bug fixes in H2O MLOps, see the H2O MLOps v0.62.0 release notes.

H2O Enterprise Steam​

  • Helm: Added annotations and labels for H2O Cloud integration.
  • Helm: Added support to override the namespace where the chart is deployed to.
  • H2O Kubernetes: Added option to specify extra environmental variables.
  • Configuration: Added deprecation mode in the licensing configuration.

AI App Store​

  • Added new fields for H2O MLOps telemetry.
  • Added support of affinity, nodeSelector, tolerations, and securityContext.
  • Provided the ability to batch messages before they are sent to message queue in asynchronous server mode.
  • Added Python CPU 3.9 and 3.10 runtimes.
  • Users can now configure the timeout duration for bundle upload.
  • Users can now configure the database connection limits.
  • Screenshots can now be sorted in lexicographic order.
  • Added the ability to emit telemetry login events on token session exchange.
  • Implemented a check for custom image for federated container apps.
  • Added the h2o admin app import cli command to enable importing apps
  • Added the h2o app import --set-image flag for setting a container image for both regular and admin users
  • Added h2o app list --precondition for filtering apps based on their precondition status checks
  • Updated the user flow for creating AI Engines.
  • Added support for the node count field for H2O3 Engines
  • Added support for searching AI Engines
  • Added the Platform Usage Page and the PlatformUsageEnabled config option for enabling the Platform Usage Page.
  • Added Admin secret management
  • Added a Runtime.AppMode container for launching docker images as apps
  • The Precondition Checker API validates whether an app is runnable and scans apps periodically for validation.
  • Added the requireRuntimeVersion config option to force apps to have a configured runtime version when importing
  • Disabled the AI Unit scanning by default in preparation for the transition to telemetry service
  • Added precondition checks for Apps to detect deprecated runtime versions
info

For a full list of features, improvements, and bug fixes in AI App Store, see the AI App Store v0.23.0, v0.24.1, v0.25.0, v0.25.1, v0.25.2, v0.25.3, v0.26.0, v0.27.0, v0.28.0, v0.28.3, v0.28.4, v0.28.6, & v0.29.1 release notes.

H2O-3​

  • Added ability to publish models to MLOps via Python API.
  • Added ability to grid over Infogram.
  • Implemented Regression Influence Diagnostics for GLM.
  • Enhanced GBM procedures to output which records are used for each tree.
  • Added learning curve plot to H2O’s Explainability.
  • Added save_plot_path parameter for Fairness plotting allowing you to save plots.
  • Added official support for Python 3.9, 3.10, and 3.11.
  • Implemented AIC metric for all GLM model families.
  • Implemented Tweedie variance power maximum likelihood estimation for GLM.
  • Added ability to convert H2OAssembly to a MOJO2 artifact.
  • Implemented new Decision Tree algorithm.
  • Enabled H2OFrame to pandas DataFrame using multi-thread from datatable to speed-up the conversion process.
  • Added support for EMR 6.10.
  • Implemented new write_checksum parameter that allows you to disable default Hadoop Parquet writer systematically writing a .crc checksum file for each written data file.
  • Implemented make_metrics with custom AUUC thresholds, and MOJO support for UpliftDRF.
  • Implemented custom metrics for:
    • AutoML
    • Stacked Ensemble
    • Deep Learning
    • UpliftDRF
    • leaderboard
  • Implemented new AdaBoost algorithm for binary classification.
  • Implemented Shapley values support for ensemble models.
info

For a full list of features, improvements, and bug fixes in H2O-3, see the H2O-3 v3.40.0.1, v3.40.0.2, v3.40.0.4, v3.42.0.1, v3.42.0.2, v3.42.0.3, v3.42.0.4, v3.44.0.1, & v3.44.0.2 release notes.

H2O Hydrogen Torch​

  • A new data connector for Google Cloud Storage is now available.
  • Supports air-gapped environments without internet access.
  • Added MLAPI support which enables you to run Hydrogen Torch UI on a CPU instance and send GPU jobs (train experiment, predict experiment) through the API to MLAPI. This is beneficial for a number of reasons including saving GPU resources.
  • Added support for .MP3 audios in MLOPs deployments
info

For a full list of features, improvements, and bug fixes in H2O-3, see the H2O Hydrogen Torch release notes.

H2O Feature Store​

  • Internal database used to store meta-data was changed from Mongo to Postgres and MongoDB collection data source introduced
  • Introduction of project history
  • Integration with H2O AI Cloud Discovery Service
  • Introduce APIs and for the following:
    • Pause and resume scheduled ingest tasks
    • Cancel a job and improve handing of cancelled jobs
    • Download a pre-generated notebook demonstrating retrieve flow
    • Upload and download artifacts to a specific feature set
    • Mark/unmark feature as target variable
    • View popular feature sets
    • View recent projects and recent feature sets
  • Ability to create feature sets, delete major feature set versions, and order projects, feature sets, or features via the GUI
  • Ability to list jobs, filter jobs based on type, and see progress of jobs via the GUI
  • Ability to create, list and revoke personal access tokens in the UI
  • Introduced approval process in CLIs and backend
  • Added support for LinkerD
  • Expose monitoring and custom data on feature schema
  • Azure Gen2 Jar is now published to maven central
  • Introduce feature set flow configuration - user can configure synchronization between online and offline stores
  • Integrate with H2O AI Cloud LoggingService
  • IAM support for Redis
  • Support for passing security context for containers
  • Ability to download pre-generated retrieve notebook via CLI and UI
  • Implemented review process in the UI
  • Ability to ingest and retrieve from UI
  • Ability for Feature Store administrator to specify maximum duration of a personal access token
  • Ability to pass affinity specification to Feature Store pods via Helm
  • Ability to obtain JDBC connection string for core PostgreSQL database from existing secret
  • Ability for namespace override in Helm
  • Add telemetry.cloud.h2o.ai/include: true annotation to Spark driver and executors
  • Add ownership attribution labels/annotations to feature store resources
info

For a full list of features, improvements, and bug fixes in H2O Feature Store, see the H2O Feature Store v0.14.3, v0.14.4, v0.15.0, v0.16.0, v0.17.0, v0.18.0, v0.18.1, v0.19.0, v0.19.1, v0.19.2, & v0.19.3 release notes.

H2O Document AI​

  • Introduced the universal scoring pipeline.
  • Introduced ability to automatically purge training artifacts.
  • Introduced ability to schedule the deletion of your whole project and all of its resources.
  • Introduced new base models for training a model in Publisher.
  • Introduced learning rate for model training in Publisher.
  • Implemented role-based access control (RBAC) to Document AI on HAIC.
info

For a full list of features, improvements, and bug fixes in H2O Document AI, see the H2O Document AI v0.7.0 release notes.

H2O Model Validation​

  • Added support for the following tests:
    • Calibration score
    • Segment performance
    • Robustness
  • Ability to generate insights for a validation test or dataset after defining a large language model (LLM) source.
  • Manage all validation tests within the Tests card.
  • Supports connections to H2O AI Engine Manager v0.5.5 or a previous version and Driverless AI (DAI) version 1.10.6.2 or a previous version.
  • Does not allow deleting models or datasets that are linked to an existing test.
  • Removed the Worker feature
  • Improvements to drift detection, segment performance tests, adversarial similarity tests, and size dependency tests.
  • Ability to view pairwise correlations between numerical columns of a dataset.
info

For a full list of features, improvements, and bug fixes in H2O Model Validation, see the H2O Model Validation v0.16.0 release notes & v0.17.0 release notes.

H2O AI Engine Manager​

  • Enabled WebSockets for DAIEngine, which resolves issues with using H2O Driverless AI Wizards.
  • Added the GPUTolerations field to DAISetup and H2OSetup.
  • Migration tool now accepts the H2O Enterprise Steam admin username and password as an alternative to PAT (Personal Access Token).
  • Added a CONNECTING state to DAIEngine.
  • Basic integration with H2O Logging Service.
  • Added registration with H2O Discovery Service.
  • Published the Python client on PyPI.
  • DAIEngine: Added support for Triton in 1.10.5.
  • DAIEngine: The engine owner is added to the local administrator list.
  • DAIEngine: The home folder is now in a writeable location and persisted across restarts.
  • DAIEngine: All entities are migrated to the engine owner during startup (>= 1.10.5).
  • DAIEngine: Added API method to change ownership of an engine and migrate the DAI entities.
  • DAIEngine: Increased the network request timeout to engines to 24 hours.
  • DAIEngine: Ensure that the engine fails when any container exits with a non-zero exit code.
  • DAIEngine: Use DAI idle indicator instead of looking at running experiments.
  • DAIEngine: Set proper override_virtual_cores and virtual_cores_per_physical_core values.
  • DAIEngine: Allow pausing engines while in CONNECTING state.
  • DAISetup: Added MaxUnusedDuration, which, when set deletes the engines when they are paused for too long.
  • H2OEngine: Added support for H2O 3.40.
  • H2OEngine: Added support for multinode clusters.
  • H2OEngine: Added support for external XGBoost.
  • Engine: Exposed current idle and running durations.
  • Engine: Added advanced filtering to support concurrent searching and filtering functionality.
  • Python: When creating an engine, the engine_id argument is no longer mandatory and will be generated if not provided.
  • Migration: Change ownership of all files to a known user during migration.
  • Helm: Added configurable Pod security context.
  • Logging: Added logging of manager events into LoggingService.
  • Logging: Added promtail sidecar for logging data from DAI filesystem into LoggingService.
  • Logging: Added creator label to scraped logs.
  • Python: Allow connecting only to running engines.
  • Telemetry: Added telemetry usage labels to DAIEngine and H2OEngine pods.
  • Added configurable SSL cert verification for the Python client.
  • Added support for Steam migrator filtering by providing a list of instances to be included or excluded from migration.
  • H2OEngine: Added API for automatic sizing.
  • Added API for DAIVersion and H2OVersion.
  • Added API for DAIProfiles.
  • Added super admin role for global resources.
  • Enabled seamless Python client login.

H2O AutoDoc​

  • Added the ability to create an AutoDoc for an H2O Driverless AI experiment in AI Engine Manager
  • The default runtime for H2O AutoDoc has been set to Python 3.8

H2O Label Genie​

  • Ability to import datasets from H2O Drive and download/export already approved samples to H2O Drive.
  • Supports multi-label for all classification annotation tasks.
  • Supports the following new annotation tasks:
    • Instance segmentation
    • Text summarization
    • Text-generative AI
  • h2oGPT and OpenAI LLMs are available for text summarization and text-generative AI annotation tasks.
  • Added several keyboard shortcuts that enable you to speed up the annotation process of a dataset. For more information, see Hotkeys
  • Added box to polygon feature that enables you to improve the efficiency of manual labeling for an image instance segmentation task.
info

For a full list of features, improvements, and bug fixes in H2O Label Genie, see the H2O Label Genie v0.3.0 release notes and v.0.4.0 release notes.

Known issues and workarounds​

Known IssueWorkaround
When adding a file in Edit in Page View on H2O Document AI, the changes are not reflected on the Annotation Set list view.Refresh the page or click on another menu item (e.g. Document Sets) and
click back to annotation sets to see the updated number of pages.
When adding a file in Edit in Page View of an Annotation Set on H2O Document AI, it only supports image files. JSON files are not supported.Export the .pdf as a .jpg or .png and then add the file in edit page view.
After adding a file in Edit in Page View in the H2O Document AI publisher, if you leave edit in page view and then return to it again, the added file does not show up.Currently, no workaround available.
H2O Document AI pipeline creation fails on OpenShift when Linkerd is enabled.Exclude port 443 in document-ai-viewer namespace by annotating the namespace with 
config.linkerd.io/skip-outbound-ports: "443"
When using the Edit in Page View in the H2O Document AI publisher on the Apple Safari browser, the first label gets pre-populated resulting in the bounding box not being labeled as such.This issue does not occur when using Google Chrome.
Keycloak pods are not started by default when Linkerd is enabled by default.Keycloak pods are not injected to the Linkerd proxy for fresh installs even
though the Linkerd inject is enabled for the Keycloak namespace.
Restart the Keycloak pod after Linkerd pods come up in your Kubernetes cluster.
Variable gcp_region does not take affect for Kubernetes module installation.Add the following line in the file terraform/templates/kubernetes/main.tf,
under the module kubernetes_gcp{}. 
gcp_region = var.gcp_region
H2O Document AI proxy ships a network policy that blocks Egress traffic from post-processor pods. If your H2O AI Hybrid Cloud environment has Calico (or any CNI that enforces NetworkPolicy) + Linkerd enabled, the custom post processor pod fails after 120 seconds.If Linkerd is enabled, delete the networkpolicy in the
document-ai-scorer namespace after deployment.
Alternatively, you can also update the helm flag as follows. 
publish: 
networkPolicy: 
enabled: false
or
publish.networkPolicy.enabled=false
Unable to start H2O-3 in H2O AI Engine Manager when Linkerd is enabled by default.Disable Linkerd in H2O AI Engine Manager. 
linkerd_aiem_manager_annotations  = { 
       "linkerd.io/inject"  =  "disabled" 
}
H2O Driverless AI versions prior to v1.10.4.2 are inaccessible post-migration to private CAUpgrade H2O Driverless AI to v1.10.4.2 or a later version.
Permission issue in container resulted in Minio job that creates the Minio bucket failing on OpenShift.Create the buckets manually
The Enterprise Steam setup fails to run on airgapped environments as it requires creating a virtual environment and installing the Steam client.Configure Enterprise Steam on the UI manually.
The mlops_helm installation uses the wrong code to get db_admin_password/db_name/db_address for deployment of PostgreSQL Server.Update lines 54, 59 and 129 on the modules/applications/aws/data.tf file
to use var.external_postgres_deployment variable instead of var.external_postgres.
When running the deployments-tf-scale-down.sh or deployments-tf-scale-up.sh scripts during the upgrade, the script fails in the middle.Replace lines 24-26 of the relevant script with the following code snippet: 
optional_components=( 
    'ui' 
    'externalregistry' 
    'prometheus-server' 
) 
 
for oc in "${optional_components 
[ 
@ 
] 
}"; do 
  if kubectl -n "$ 
ns" get deployment " 
${prefix}-${oc}" >/dev/null 2> 
& 
1; then 
    components+=("${oc}") 
  fi 
done
The variable name used to define the H2O MLOps monitoring db name is mlops_monitoring_db but in dependencies it is named as mlops_monitoring_db_name.
The same variable name should be used to avoid any issues.		Include both variables in the config.tfvars file to avoid any issues
Restoring the external-registry-adapter fails.Update the following variables with the following values in templates and modules:
  • mlops_monitoring_db_username = "mlops_model_monitoring"
  • mlops_deployment_db_username = "deployment_server"
  • mlops_db_username = "h2oai_storage"
  • mlops_deployment_db_name = "deployment_server"
  • mlops_monitoring_db = "mlops_model_monitoring"
  • mlops_db_name = "h2oai_storage"
The image pull secret does not get created in the feature-store-redis namespace due to the namespace variable being empty.Include featurestore_redis_namespace = "feature-store-redis" in the config.tfvars file or update templates & modules to set feature-store-redis to the variable.

v23.01.2 | Aug 23, 2023​

Important

End of support date: April 14, 2024.

For more information, see About version support.

Upgraded components​

Optional components​

Optional components of HAIC are provided to customers as needed based on your specific requirements. If you have any of the following components, note the following upgrades.

  • H2O Document AI v0.6.2

New features​

  • Added the ability to update or add new templates in H2O Document AI

Bug fixes​

  • Fixed an issue in H2O Document AI where the logic used to pull the feature store web proxy image did not work if the image registry is empty.
  • Made an improvement to include input_dir content in the request going to the custom post-processor deployment.

Known issues and workarounds​

Known IssueWorkaround
Deploying an H2O Document AI pipeline with a long name fails.To avoid this issue, use pipeline names that are as short as possible (preferably 3-4 characters) in H2O Document AI. In a future release, the issue will be fixed by decoupling the pipeline resource names from the name provided by the customer.
After publishing a pipeline using a model trained on one file on H2O Document AI Viewer, the document will show zero results.This has more to do with the quality of the model trained. If you try to score with the same file used for training then it will probably, though not guaranteed, find at least one entity.
Additionally, it also depends on how the bounding box was created. If the bounding box is very small or useless, then it may result in this situation.
H2O Document AI proxy ships a network policy that blocks Egress traffic from post-processor pods. If your H2O AI Hybrid Cloud environment has Calico (or any CNI that enforces NetworkPolicy) + Linkerd enabled, the custom post processor pod fails after 120 seconds.If Linkerd is enabled, delete the networkpolicy in the document-ai-scorer namespace after deployment. Alternatively, you can also update the helm flag as follows.
publish:
networkPolicy:
enabled: false

or publish.networkPolicy.enabled=false

v23.01.1 | Jul 28, 2023​

Important

End of support date: April 14, 2024

For more information, see About version support.

New components​

H2O LLM Studio is an optional component of new installations of HAIC.

note

Let your H2O contact know if you would like to install this beta application into your AI App Store.

Upgraded components​

Core components​

  • H2O Driverless AI v1.10.5
  • H2O MLOps v0.61.1
  • Enterprise Steam v1.9.5

Optional components​

Optional components of HAIC are provided to customers as needed based on your specific requirements. If you have any of the following components, note the following upgrades.

  • H2O Hydrogen Torch v1.3.0
  • H2O Document AI v0.6.1
  • H2O AutoInsights v0.7.3
  • H2O Model Analyzer v0.7.2
  • H2O Model Validation v0.15.2

New features​

Platform installer

  • Upgraded Amazon EBS CSI from v2.4.1 to v2.17.1
  • Added metrics server to Amazon EKS
  • Added support for Linkerd on Azure and OpenShift
  • Added S3 for AI App Store on OpenShift
  • Replaced the MLOps legacy UI with the Wave UI
  • Upgraded Amazon RDS versions to v11.20 on AWS
  • Telemetry is upgraded to v0.25
  • Compatibility up to Kubernetes v1.26

H2O Driverless AI

  • Added support for NVIDIA Triton Inference Server (disabled by default and should remain disabled for H2O AI Hybrid Cloud Openshift customers)
  • Several improvements to the Experiment Wizard
  • Added an Experiment Results Wizard (beta) and an Experiment Comparison Wizard
  • Added CPU memory usage of C++ MOJO to experiment summary and to Deploy Wizard.
  • Added a page that lets admin users view system logs.
  • Added MLI python client along with explanatory videos for MLI explainers
  • Added support to transform a dataset with the experiment’s fitted pipeline
  • Numerous bug fixes
  • Several UI/UX and performance improvements
    info

    For a full list of features, improvements, and bug fixes in H2O Driverless AI, see the H2O Driverless AI v1.10.5 release notes.

H2O MLOps

  • Introduced a feature flag to enable the import third-party experiments (pickled experiments) flow with Conda.
  • Users can now create A/B Test and Champion/Challenger deployments through the UI.
  • Users can now create and view configurable scoring endpoints through the UI.
  • Concurrent Scoring Requests are now supported for Python-based Scorers. Scoring times for C++ MOJO, Scoring Pipeline, and MLflow types now support parallelization with the default degree of parallelization set to 2.
  • Support for H2O3 MLflow Flavors and import of MLflow wrapped H2O3 models.
  • Added support for Kubernetes v1.26
  • Added full support for DAI v1.10.5
  • Added support for the H2O discovery service.
  • Added loglevel propagation for the deployment and scoring pipeline and increased the default loglevel from DEBUG/TRACE to INFO.
  • Bug fixes to the deployment pipeline, monitoring, and drift detection.
  • Add mutually exclusive lock protection when multiple update deployment requests are received against the same target deployment.
  • The standalone version of the wave app is now fully air-gap compatible.
    info

    For a full list of features, improvements, and bug fixes in H2O MLOps, see the MLOps v0.60.1 & v0.61.1 release notes.

H2O Hydrogen Torch

  • Added support for the following problem types:
    • Speech recognition
    • 3D image classification
    • 3D image regression
    • 3D image semantic segmentation
  • Users can now deploy built models to H2O MLOps directly from the H2O Hydrogen Torch UI.
  • The .whl package of a Python scoring pipeline now includes a Dockerfile that enables users to build a dedicated Docker image that can include all requirements to run the scoring pipeline for model inference (production).
  • The H2O Hydrogen Torch landing page now has a new design that provides an array of statistics and facts about the instance of the application.
  • Several new UX improvements to improve user experience.
    info

    For a full list of features, improvements, and bug fixes in H2O Hydrogen Torch, see the Hydrogen Torch v1.3.0 release notes.

H2O Document AI

  • Introduced H2O Document AI - Viewer for business users to score documents on built pipelines.
  • Introduced initial telemetry integration.
  • Added the ability to score PDFs with page ranges.
  • Added support for Kubernetes 1.26.
  • Updated telemetry implementation to make scored documents more efficiently retrieved. ::info For a full list of features, improvements, and bug fixes in H2O Document AI, see the Document AI v0.6 release notes. :::

H2O AutoInsights

  • Support for basic Treemap chart for hierarchical category analysis
  • Enable ability to download flagged anomalies (Multivariate)

H2O Model Analyzer

  • DAI 2023 License update
  • Minor UI/UX improvements

Enterprise Steam

  • Kubernetes: Added support for Kubernetes 1.26
  • Kubernetes: Configurable seccomp profile
  • Helm: Fixed support for custom CA certificates
  • Helm: Removed unnecessary RBAC permissions
  • Added support for the H2O AI Cloud Discovery Service
  • H2O Driverless AI: Added support for Driverless AI 1.10.5
  • H2O Driverless AI: Automatic entity migration when changing the ownership on an instance
  • H2O Driverless AI: Added option to toggle Triton (and CAP_SYS_NICE capability) on/off
  • Python: Reduced excessive logging of client requests
  • Hadoop: Added support for Hadoop 3.3
    info

    For a full list of features, improvements, and bug fixes in Enterprise Steam, see the Enterprise Steam v1.9.1, v1.9.2, v1.9.3, v1.9.4, & v1.9.5 release notes.

Known issues and workarounds​

Known IssueWorkaround
H2O Document AI pipeline creation fails on OpenShift when Linkerd is enabled.Exclude port 443 in document-ai-viewer namespace by annotating the namespace with config.linkerd.io/skip-outbound-ports: "443".
Keycloak pods are not started by default when Linkerd is enabled by default.Keycloak pods are not injected to the Linkerd proxy for fresh installs even though the Linkerd inject is enabled for the Keycloak namespace. Restart the Keycloak pod after Linkerd pods come up in your Kubernetes cluster.
Variable gcp_region does not take affect for Kubernetes module installation.Add the following line in the file terraform/templates/kubernetes/main.tf, under the module kubernetes_gcp{}.

gcp_region = var.gcp_region
Typo in the image name for H2O DocumentAI Viewer.Set a value for image_registry on your bundle.
Unable to start H2O DriverlessAI wizard from AI Engine Manager.Fixed in AI Engine Manager v0.2.3. Optionally, you can use Enterprise Steam if the nitro wizard is required.
Deploying an H2O Document AI pipeline with a long name fails.To avoid this issue, use pipeline names that are as short as possible (preferably 2-3 characters) in H2O Document AI. In a future release, the issue will be fixed by decoupling the pipeline resource names from the name provided by the customer.
When adding a file to an annotation set on H2O Document AI publisher in Edit in Page View, the number of documents and pages may not get properly updated.Refresh the page, or click on another menu item (e.g. Document Sets) and then click back to Annotation Sets to see the updated number of pages.
Only image files are supported when adding a file in Edit in Page View in the H2O Document AI publisher.To add a file that is not an image, first export the PDF as a JPG or PNG. Then, add the file in Edit in Page View.
After adding a file in Edit in Page View in the H2O Document AI publisher, if you leave edit in page view and then return to it again, the added file does not show up.Currently, no workaround available.
When using the Edit in Page View in the H2O Document AI publisher on the Apple Safari browser, the first label gets pre-populated resulting in the bounding box not being labeled as such.This issue does not occur when using Google Chrome.
If the image registry is empty, the logic used to pull feature store web proxy image will not work.Change line 11 of the file terraform/modules/applications/featurestore/locals.tf to the following:

featurestore_web_proxy_image = var.image_registry != "" ? "${var.image_registry}/${var.featurestore_web_proxy_image}" : "gcr.io/vorvan/h2oai/${var.featurestore_web_proxy_image}"

v23.01.0 | Apr 14, 2023​

Important

End of support date: April 14, 2024

For more information, see About version support.

New components​

AI Engine Manager (AIEM) is an optional component of new installations of HAIC, and is recommended if you are using any of the following product versions:

  • H2O Driverless AI v1.10.4 or a later version
  • H2O-3 v3.38.0.4, v3.38.0.3, or v3.36.1.5

However, if you are currently using, or wish to use an older version of H2O Driverless AI or H2O-3, it is recommended to use Enterprise Steam instead of AI Engine Manager.

The goal of AIEM is to be the central service integrating with Kubernetes for model building tools like H2O Driverless AI and H2O-3, enabling other teams to focus on making the best ML products instead of managing Kubernetes resources. AIEM also features an interface found directly within the HAIC cloud interface to improve user flow.

Upgraded components​

Core components​

  • AI App Store v0.22.0
  • H2O Driverless AI v1.10.4.2 & v1.10.4.3
  • H2O-3 v3.38.0.3 & v3.38.0.4
  • H2O MLOps v0.59.0, 0.59.1, & v0.60.1
  • Enterprise Steam v1.9.0

Optional components​

Optional components of HAIC are provided to customers as needed based on your specific requirements.

  • H2O Feature Store v0.14.4 is currently only available as a preview for H2O Hybrid Cloud users on AWS and Azure.
  • H2O Document AI v0.5.0 provides automated document processing and extracts insights from text, images, and tables. For more information, see the H2O Document AI release notes.
  • H2O Hydrogen Torch v1.2.0 is an application for training and deployment of deep learning models.

New features​

AI App Store

  • Improvements to the 'My Apps' page:
    • Added support for Python Apps
    • Easily manage apps from the UI
    • Find the app you’re looking for more easily
    • Easily copy the UUID
  • Improvements to the 'My Instances' page:
    • Find instances more easily
    • Quickly find all versions of the exact app you’re seeking
    • Enhanced app filtering
  • Improvements to the 'Admin Apps' page:
    • Find the app you’re looking for more easily
    • Easily copy the UUID
  • Improvements to the 'Admin Instances' page:
    • Sort app instances in the admin instances UI
    • Improve scalability of app instances

Document AI

  • Added optical character recognition (OCR) language support for:
    • Latin (e.g. Spanish)
    • Arabic (e.g. Persian)
  • Added Document Text Recognition (DocTR) EfficientNet models to better recognize handwritten documents
  • Added ability to set batch size and number of epochs for model training
  • Upgraded the ML API to v0.4.0
  • Refactored and improved the training user interface for better usability
  • Added role based access control (RBAC) for Doc AI visibility in HAIC to gate access to H2O Document AI based on a user's role
  • Added command-line bulk scorer to score a large number of documents (optional feature and shipped separately from the main product)

H2O Driverless AI

Hybrid Cloud integration & deployment

  • MLOps/Remote Storage integration:
    • Added functionality in the Python/R API and Flow to push an H2O-3 model to Remote Storage/MLOps
  • Added automated testing:
    • Environments for fresh install testing
      • v23.01.0 on AWS
      • v23.01.0 on Azure
      • v23.01.0 on GCP
    • Environments for upgrade testing to v23.01.0
      • v22.10.1 on AWS, GCP, and Azure
      • v22.10.0 on AWS
      • v22.07.2 on AWS
      • v22.04.1 on AWS
  • Modular auto-installer
    • Integrate Document AI to deployment templates
    • Integrate Feature Store to deployment templates
    • Support for Kubernetes v1.23
    • Support for Kubernetes v1.24
    • Ongoing Improvements based on v22.10.0 installation/upgrade feedback from the field

H2O MLOps

  • Added static scoring endpoints
  • Supports downloading deployment logs from the MLOps Wave App and MLOps API
  • Runtime support for H2O Driverless AI v1.10.4.1, v1.10.4.2, & v1.10.4.3
  • Replaces a Kubernetes configmap with a Kubernetes secret
    info

    For a full list of features, improvements, and bug fixes in MLOps, see the MLOps v0.60.1, 0.59.0 & v0.59.1 release notes.

H2O-3

  • Implemented p-value calculation for GLM with regularization
  • Implemented normal (non-monotonic) splines that can support any degrees
  • Verified the minimum number of knots each spline type can support for GAM
    info

    For a full list of features, improvements, and bug fixes in H2O-3, see the H2O-3 v3.38.0.3 & v3.38.0.4 release notes.

Platform UI

  • Access AI Engines from the Hybrid Cloud UI

Enterprise Steam

  • Driverless AI: Added support for single-user multinode clusters and fixed maximum uptime detection
  • Sparkling Water: Added support for the latest version of Sparkling Water
  • Kubernetes: Disabled auto-mounting of service account tokens
  • Kubernetes: Set specific secure default for Pod seccomp profile
  • Python: Removed tokens from Python client logging
  • Helm: Added support for custom CA certificates
    info

    For a full list of features, improvements, and bug fixes in Enterprise Steam, see the Enterprise Steam v1.9.0 release notes.

v22.10.1 | Dec 19, 2022​

Important

End of support date: November 16, 2023

For more information, see About version support.

Component upgrades​

  • H2O MLOps v0.58.0

New features​

Kubernetes 1.23 support

H2O AI Cloud now supports Kubernetes 1.23 for the following deployments:

  • Amazon AWS EKS
  • Google GCP GKE
  • Microsoft Azure AKS

H2O MLOps

  • Added support for linking and deploying H2O Driverless AI unsupervised models

  • H2O Driverless AI flavours for MLflow models are now supported through integration with Azure Databricks

  • H2O eScorer integration to support batch deployment workflows

  • UX optimizations and additional improvements to the H2O MLOps Wave app

  • Fixed various issues with the H2O MLOps admin dashboard

  • Shapley values can now be calculated for H2O Driverless AI Python pipelines and MOJOs

  • Improvements and bug fixes related to Python scoring pipelines runtimes

info

For a full list of features, improvements, and bug fixes in H2O MLOps, see the H2O MLOps v0.58.0 release notes.

Installer

  • Improvements and bug fixes to the HAIC installer
  • Resolved incompatibility with newer versions of the ebs-csi driver on AWS

H2O Driverless AI

  • Added the H2O MLOps v1.10.3.2 runtime

Known limitations​

H2O Driverless AI

Note that the H2O Driverless AI 1.10.4 Python client is not backwards-compatible with H2O Driverless AI v1.10.3.2. Instead, you can use this v1.10.3.2 Python client with DAI v1.10.3.2.

The DAI v1.10.4.1 Python client includes backward-compatibility for v1.10.3.2 and will be available with the upcoming H2O AI Cloud v23.01.0 release.

Release FAQ​

Do I need to upgrade to v22.10.1?

Upgrading is recommended if you are on v22.10.0 and need H2O MLOPs v0.58.0 or Kubernetes 1.23.

I am currently using a version prior to v22.10.0. Can I directly upgrade to this version without going to v22.10.0?

Yes. You can upgrade from v22.07.0 or v22.04.0 to v22.10.1.

When v23.01.0 is released, will I need to first upgrade to v22.10.1, or can I upgrade directly to v23.01.0?

Depending on the version currently installed, the environment may have to be upgraded to an intermediate version before v23.01.0 can be installed. However, v22.10.1 specifically may not be required in the migration path. For more information about H2O AI Cloud upgrade paths, contact your H2O.ai support team.

v22.10.0 | Nov 16, 2022​

Important

End of support date: November 16, 2023

For more information, see About version support.

Component upgrades​

  • AI App Store v0.21.0
  • Driverless AI v1.10.4
  • Hydrogen Torch v1.2.0
  • H2O-3 v3.36.1.5
  • H2O MLOps v0.57.3
  • Enterprise Steam v1.8.14

New features​

AI App Store

  • Developers can now retrieve the app.toml file and app bundles from the server using H2O CLI commands.

  • Support for Python 3.8 runtimes

Driverless AI

  • New GUI-based wizards to configure and start experiments, join datasets, and perform business value analyses.

  • Improvements in data handling and detection.

  • New BinnerTransformer for one-dimensional binning of numeric features.

  • Support for prediction intervals for regression experiments in Java MOJO scoring (for both C++ and Java MOJO runtimes).

  • New feature_store_mojo recipe type to create a MOJO to be used as a feature engineering pipeline in the H2O Feature Store.

  • Users can now run Disparate Impact Analysis on external datasets.

  • Users can now bulk abort multiple experiments in a project.

    info

    For a full list of features, improvements, and bug fixes in Driverless AI, see the Driverless AI v1.10.4 release notes.

Hydrogen Torch

  • New Azure Data Lake data connector

  • Support for several dataset (data) formats for:

    • image object detection experiments
    • image semantic segmentation experiments
    • image instance segmentation experiments

  • Users can now merge imported datasets into one as well as extend datasets with new data.

  • Users can organize experiments into projects and compare experiments in a project.

  • Improved experiment settings and support for several grid search modes.

    info

    For a full list of features, improvements, and bug fixes in Hydrogen Torch, see the Hydrogen Torch v1.2.0 release notes.

H2O-3

H2O MLOps

Enterprise Steam

  • Users are now able to change the ownership of Driverless AI instances.

  • Bug fixes related to updating user roles and session ready time.

    info

    For a full list of features, improvements, and bug fixes in Enterprise Steam, see the Enterprise Steam v1.18.4 release notes.

Installer

  • Restructured installer and modules

  • CI/CD pipeline for deployment package automation

  • Upgrade guide for existing customers

  • OpenShift support for Hybrid Cloud and updated Azure platform support

Feedback