Scoring runtimes
This page describes the scoring runtimes available for model deployment, including configuration options and usage instructions for each runtime type.
Runtime options
The selection of available runtimes is determined by the artifact type that you specify. The following list provides information on the available options when selecting an artifact type and runtime.
Selecting an incorrect runtime causes the deployment to fail.
Artifact type | Runtime option(s) | Notes |
---|---|---|
Driverless AI MOJO pipeline/ MLflow Driverless AI MOJO pipeline | Driverless AI MOJO Scorer Driverless AI MOJO Scorer (Shapley original only) Driverless AI MOJO Scorer (Shapley transformed only) Driverless AI MOJO Scorer (Shapley all) Driverless AI MOJO Scorer (C++ Runtime) | • Original only and Transformed only require 2× memory vs. Shapley none. • Shapley all requires 3× memory. • C++ Runtime requires experiment to be linked through workspace. |
Driverless AI Python pipeline/ MLflow Driverless AI Python pipeline | Python Pipeline Scorer [Driverless AI 1.10.7–1.11.1.1, 2.0.0–2.2.0] | Python scorer version must match the Driverless AI version used to build the model. |
H2O-3 MOJO/ MLflow H2O-3 MOJO | H2O-3 MOJO Scorer H2O-3 MOJO Scorer (Shapley transformed only) | |
MLflow zipped | [PY-3.10][CPU] HT Flexible Runtime [PY-3.10][GPU] HT Flexible Runtime [Py 3.9–3.12] Dynamic MLflow Model Scorer | For usage details, see MLflow Dynamic Runtime. |
-
The C++ MOJO2 runtime (
Driverless AI MOJO Scorer (C++ Runtime)
) accepts a wider range of algorithms Driverless AI may use that the Java runtime does not support, including BERT, GrowNet, and TensorFlow models. If you want to use one of these models, it must be linked from Driverless AI and not be manually uploaded. -
MLflow runtimes support Python 3.9 and later.
-
For end of support information on H2O Driverless AI runtimes, see the Driverless AI Prior Releases page.
Artifact names mapping
The following table describes the mapping of artifact names.
Artifact type name | Storage artifact type | Artifact type |
---|---|---|
DAI MOJO Pipeline | dai/mojo_pipeline | dai_mojo_pipeline |
DAI Python Pipeline | dai/scoring_pipeline | dai_python_scoring_pipeline |
H2O-3 MOJO | h2o3/mojo | h2o3_mojo |
MLflow zipped | python/mlflow | python/mlflow.zip |
MLflow DAI MOJO Pipeline | mlflow/mojo_pipeline | mlflow_mojo_pipeline |
MLflow DAI Python Pipeline | mlflow/scoring_pipeline | mlflow_scoring_pipeline |
MLflow H2O-3 MOJO | mlflow/h2o3_mojo | mlflow_h2o3_mojo |
Runtime names mapping
The following table describes the mapping of runtime names.
Compatible model | Runtime name | Runtime |
---|---|---|
Driverless AI MOJO models (Java runtime) | Driverless AI MOJO Scorer | dai_mojo_runtime |
Driverless AI MOJO models (C++ runtime) - supports all Shapley contribution types and is expected to have significantly lower memory usage | Driverless AI MOJO Scorer (C++ Runtime) | dai-mojo-cpp_experimental |
Driverless AI MOJO models (Java runtime) - with Shapley contributions for original features | Driverless AI MOJO Scorer (Shapley original only) | mojo_runtime_shapley_original |
Driverless AI MOJO models (Java runtime) - with Shapley contributions for transformed features | Driverless AI MOJO Scorer (Shapley transformed only) | mojo_runtime_shapley_transformed |
Driverless AI MOJO models (Java runtime) - with Shapley contributions for both original and transformed features | Driverless AI MOJO Scorer (Shapley all) | mojo_runtime_shapley_all |
Driverless AI Python Scoring Pipeline models created by Driverless AI 1.10.7 | Python Pipeline Scorer [Driverless AI 1.10.7] | python-scorer_dai_pipelines_1107 |
Driverless AI Python Scoring Pipeline models created by Driverless AI 1.10.7.1 | Python Pipeline Scorer [Driverless AI 1.10.7.1] | python-scorer_dai_pipelines_11071 |
Driverless AI Python Scoring Pipeline models created by Driverless AI 1.10.7.2 | Python Pipeline Scorer [Driverless AI 1.10.7.2] | python-scorer_dai_pipelines_11072 |
Driverless AI Python Scoring Pipeline models created by Driverless AI 1.10.7.3 | Python Pipeline Scorer [Driverless AI 1.10.7.3] | python-scorer_dai_pipelines_11073 |
Driverless AI Python Scoring Pipeline models created by Driverless AI 1.11.0 | Python Pipeline Scorer [Driverless AI 1.11.0] | python-scorer_dai_pipelines_1110 |
Driverless AI Python Scoring Pipeline models created by Driverless AI 1.11.1.1 | Python Pipeline Scorer [Driverless AI 1.11.1.1] | python-scorer_dai_pipelines_11111 |
Driverless AI Python Scoring Pipeline models created by Driverless AI 2.0.0 | Python Pipeline Scorer [Driverless AI 2.0.0] | python-scorer_dai_pipelines_200 |
Driverless AI Python Scoring Pipeline models created by Driverless AI 2.1.0 | Python Pipeline Scorer [Driverless AI 2.1.0] | python-scorer_dai_pipelines_210 |
Driverless AI Python Scoring Pipeline models created by Driverless AI 2.2.0 | Python Pipeline Scorer [Driverless AI 2.2.0] | python-scorer_dai_pipelines_220 |
H2O-3 MOJO models | H2O-3 MOJO Scorer | h2o3_mojo_runtime |
H2O-3 MOJO models | H2O-3 MOJO Scorer (Shapley transformed only) | h2o3_mojo_runtime_shapley_transformed |
H2O Hydrogen Torch MLflow models | [PY-3.10][CPU] HT Flexible Runtime | python-scorer_hydrogen_torch_cpu_py310 |
H2O Hydrogen Torch MLflow models | [PY-3.10][GPU] HT Flexible Runtime | python-scorer_hydrogen_torch_gpu_py310 |
MLFlow non-H2O.ai models created with Python 3.9 | [Py 3.9] Dynamic MLflow Model Scorer | python-scorer_mlflow_dynamic_39 |
MLFlow non-H2O.ai models created with Python 3.10 | [Py 3.10] Dynamic MLflow Model Scorer | python-scorer_mlflow_dynamic_310 |
MLFlow non-H2O.ai models created with Python 3.11 | [Py 3.11] Dynamic MLflow Model Scorer | python-scorer_mlflow_dynamic_311 |
MLFlow non-H2O.ai models created with Python 3.12 | [Py 3.12] Dynamic MLflow Model Scorer | python-scorer_mlflow_dynamic_312 |
MLflow Dynamic Runtime
The MLflow Dynamic Runtime lets you deploy MLflow models with diverse dependencies in H2O MLOps. The following steps describe how to deploy a dynamic MLflow runtime deployment in H2O MLOps.
For an example of how to train a dynamic runtime, see Train a dynamic runtime.
-
Save your model using the
mlflow.pyfunc.save_model
function call. Use thepip_requirements
parameter to specify the Python package dependencies required by the model.mlflow.pyfunc.save_model(
path=...,
python_model=...,
artifacts=...,
signature=...,
pip_requirements=..., # <- Use this parameter to override libs for dynamic runtime
) -
After saving the model, create a zip archive of the saved model directory. Ensure that a requirements file (
requirements.txt
) that lists all dependencies is included in the zip archive. The following is an example of the expected structure for the zip file from a TensorFlow model:tf-model-py310
├── MLmodel
├── artifacts
│ └── tf.h5
├── conda.yaml
├── python_env.yaml
├── python_model.pkl
└── requirements.txt -
Depending on whether you are using Python 3.9, Python 3.10, Python 3.11, or Python 3.12 select from one of the following options:
- [PY-3.9] MLflow Dynamic Model Scorer
- [PY-3.10] MLflow Dynamic Model Scorer
- [PY-3.11] MLflow Dynamic Model Scorer
- [PY-3.12] MLflow Dynamic Model Scorer
The MLflow Dynamic Runtime has a fixed MLflow dependency, which is MLflow 2.14.2. This means that the MLflow Dynamic Runtime is not guaranteed to work with a different version of MLflow model.
Example: Train a dynamic runtime model
The following example demonstrates how to train a dynamic runtime with TensorFlow:
# Import libraries
import mlflow
import pandas as pd
import shutil
import tensorflow as tf
from sklearn import datasets
# Load and prepare data
diabetes = datasets.load_diabetes()
X = diabetes.data[:, 2:3] # Use only one feature for simplicity
y = diabetes.target
# Build and train TensorFlow model
tf_model = tf.keras.models.Sequential([
tf.keras.layers.Dense(1, input_dim=1)
])
tf_model.compile(optimizer='adam', loss='mean_squared_error')
tf_model.fit(X, y, epochs=10)
tf_model_path = "tf.h5"
tf_model.save(tf_model_path, save_format="h5")
# Enable the TensorFlow model to be used in the Pyfunc format
class PythonTFmodel(mlflow.pyfunc.PythonModel):
def load_context(self, context):
import tensorflow as tf
self.model = tf.keras.models.load_model(context.artifacts["model"])
def predict(self, context, model_input):
tf_out = self.model.predict(model_input)
return pd.DataFrame(tf_out, columns=["db_progress"])
# Generate signature from your model definition
model = PythonTFmodel()
context = mlflow.pyfunc.PythonModelContext(model_config=dict(), artifacts={"model": tf_model_path})
model.load_context(context)
x = pd.DataFrame(X, columns=["dense_input"])
y = model.predict(context, x)
signature = mlflow.models.signature.infer_signature(x, y)
# Specify a file path where the model will be saved
mlflow_model_path = "./tf-model-py310"
# Save model using MLflow
mlflow.pyfunc.save_model(
path=mlflow_model_path,
python_model=PythonTFmodel(),
signature=signature,
artifacts={"model": tf_model_path},
pip_requirements=["tensorflow"]
)
# Package model as a zip archive
shutil.make_archive(
mlflow_model_path, "zip", mlflow_model_path
)
The following is the structure of the zip file that is generated in the preceding example:
tf-model-py310
├── MLmodel
├── artifacts
│ └── tf.h5
├── conda.yaml
├── python_env.yaml
├── python_model.pkl
└── requirements.txt
Generic Ephemeral volumes
The custom additional volumes feature now supports emptyDir
volumes and ephemeral
volumes.
The storageClassName
property for volumes is optional. If not provided, the default storage class will be used.
Example configuration
# Custom additional volumes with selected mount paths.
# This section, as well as each of its fields, is optional.
volume-mounts = [
{
name = "ephemeral_volume"
type = "ephemeral"
properties = [
{ name = "size", value = "1Gi" }
]
paths = ["/ephemeral_volume_1", "/ephemeral_volume_2"]
},
{
name = "emptyDir_volume"
type = "emptyDir"
properties = [
{ name = "medium", value = "Memory" }
]
paths = ["/emptyDir_volume_1", "/emptyDir_volume_2"]
}
]
YAML configuration
The volumeMounts
section should be added to the runtime specification of the Helm Chart.
runtimes:
volumeMounts:
- name: "dev-shm"
type: "ephemeral"
properties:
size: "1Gi"
paths: ["/tmp"]
- Submit and view feedback for this page
- Send feedback about H2O MLOps to cloud-feedback@h2o.ai