Skip to main content
Version: Next 🚧

Scoring runtimes

This page describes the scoring runtimes available for model deployment, including configuration options and usage instructions for each runtime type.

Runtime options​

The selection of available runtimes is determined by the artifact type that you specify. The following list provides information on the available options when selecting an artifact type and runtime.

note

Selecting an incorrect runtime causes the deployment to fail.

Artifact typeVersionRuntime optionNotes
Driverless AI MOJO pipelineDAI 1.10.5 and laterDAI MOJO Scorer (Shapley none)
Driverless AI MOJO pipelineDAI 1.10.5 and laterDAI MOJO Scorer (Shapley original only)Requires 2x the memory as the Shapley none option.
Driverless AI MOJO pipelineDAI 1.10.5 and laterDAI MOJO Scorer (Shapley transformed only)Requires 2x the memory as the Shapley none option.
Driverless AI MOJO pipelineDAI 1.10.5 and laterDAI MOJO Scorer (Shapley all)Requires 3x the memory as the Shapley none option.
Driverless AI MOJO pipelineDAI 1.10.5 and laterDAI MOJO Scorer (C++ Runtime)Experiment needs to be linked through project.
Driverless AI Python scoring pipelineDAI 1.10.5 and laterPython Pipeline Scorer [DAI 1.10.5], Python Pipeline Scorer [DAI 1.10.5.1], Python Pipeline Scorer [DAI 1.10.6], Python Pipeline Scorer [DAI 1.10.6.1], Python Pipeline Scorer [DAI 1.10.6.2], Python Pipeline Scorer [DAI 1.10.7], Python Pipeline Scorer [DAI 1.10.7.1], Python Pipeline Scorer [DAI 1.10.7.2], Python Pipeline Scorer [DAI 1.11.0], and Python Pipeline Scorer [DAI 1.11.1]Python pipeline scorer’s version must correspond to the DAI version used to build the model (for example, a model built with DAI 1.10.7 must use Python Pipeline Scorer [DAI 1.10.7]).
H2O-3 MOJOAll versionsH2O-3 MOJO Scorer
MLflow / .pkl fileMLflow Model Scorer [Python 3.9]MLflow Model Scorer’s version must correspond to the Python version used to build the model.
MLflow[PY-3.9] MLflow Dynamic Model Scorer, and [PY-3.10] MLflow Dynamic Model ScorerFor information on how to use the dynamic runtime, see MLflow Dynamic Runtime.
note
  • The C++ MOJO2 runtime (DAI MOJO Scorer (C++ Runtime)) accepts a wider range of algorithms DAI may use that the Java runtime does not support, including BERT, GrowNet, and TensorFlow models. If you want to use one of these models, it must be linked from DAI and not be manually uploaded.

  • MLflow runtimes support Python 3.9 and later.

  • For end of support information on H2O Driverless AI runtimes, see the Driverless AI Prior Releases page.

Artifact names mapping​

The following table describes the mapping of artifact names.

Storage artifact namedeployable_artifact_type_nameArtifact processor name
dai/mojo_pipelinedai_mojo_pipelinedai_mojo_pipeline_extractor
dai/scoring_pipelinedai_python_scoring_pipelineartifact-processor_dai_pipelines_193
h2o3/mojoh2o3_mojoh2o3_mojo_extractor
python/mlflowpython/mlflow.zipunzip_processor
mlflow/mojo_pipelinemlflow_mojo_pipelinemlflow_mojo_pipeline_extractor
mlflow/scoring_pipelinemlflow_scoring_pipelinemlflow_scoring_pipeline_extractor
mlflow/h2o3_mojomlflow_h2o3_mojomlflow_h2o3_mojo_extractor

Runtime names mapping​

The following table describes the mapping of runtime names.

Model typeModel descriptionHuman-readable runtime nameRuntime name
dai_mojoDAI MOJO models (C++ runtime) - supports all Shapley contribution types and is expected to have significantly lower memory usageDAI MOJO Scorer (C++ Runtime)dai-mojo-cpp_experimental
dai_mojoDAI MOJO models (Java runtime)H2O.ai MOJO scorerdai_mojo_runtime
dai_mojoDAI MOJO models (Java runtime) - with Shapley contributions for original featuresDAI MOJO Scorer (Shapley original only)mojo_runtime_shapley_original
dai_mojoDAI MOJO models (Java runtime) - with Shapley contributions for transformed featuresDAI MOJO Scorer (Shapley transformed only)mojo_runtime_shapley_transformed
dai_mojoDAI MOJO models (Java runtime) - with Shapley contributions for both original and transformed featuresDAI MOJO Scorer (Shapley all)mojo_runtime_shapley_all
dai_python_scoring_pipelineDAI Python Scoring Pipeline models created by DAI 1.10.7Python Pipeline Scorer [DAI 1.10.7]python-scorer_dai_pipelines_1107
dai_python_scoring_pipelineDAI Python Scoring Pipeline models created by DAI 1.10.7.1Python Pipeline Scorer [DAI 1.10.7.1]python-scorer_dai_pipelines_11071
dai_python_scoring_pipelineDAI Python Scoring Pipeline models created by DAI 1.10.7.2Python Pipeline Scorer [DAI 1.10.7.2]python-scorer_dai_pipelines_11072
dai_python_scoring_pipelineDAI Python Scoring Pipeline models created by DAI 1.10.7.3Python Pipeline Scorer [DAI 1.10.7.3]python-scorer_dai_pipelines_11073
dai_python_scoring_pipelineDAI Python Scoring Pipeline models created by DAI 1.11.0Python Pipeline Scorer [DAI 1.11.0]python-scorer_dai_pipelines_1110
dai_python_scoring_pipelineDAI Python Scoring Pipeline models created by DAI 1.11.1.1Python Pipeline Scorer [DAI 1.11.1.1]python-scorer_dai_pipelines_11111
mlflowMLFlow non-H2O.ai models created with Python 3.10[PY-3.10][CPU] HT Flexible Runtimepython-scorer_hydrogen_torch_cpu_py310
mlflowMLFlow non-H2O.ai models created with Python 3.10[PY-3.10][GPU] HT Flexible Runtimepython-scorer_hydrogen_torch_gpu_py310
h2o3_mojoH2O-3 MOJO modelsH2O.ai MOJO scorerh2o3_mojo_runtime
mlflowMLFlow non-H2O.ai models created with Python 3.9MLflow Model Scorer [Python 3.9]python-scorer_mlflow_39
mlflowMLFlow non-H2O.ai models created with Python 3.10[Py-3.10] MLflow Model Scorerpython-scorer_mlflow_310
mlflowMLFlow non-H2O.ai models created with Python 3.11[Py-3.11] MLflow Model Scorerpython-scorer_mlflow_311
mlflowMLFlow non-H2O.ai models created with Python 3.12[Py-3.12] MLflow Model Scorerpython-scorer_mlflow_312
mlflowMLFlow non-H2O.ai models created with Python 3.9[Py-3.9] Dynamic MLflow Model Scorerpython-scorer_mlflow_dynamic_39
mlflowMLFlow non-H2O.ai models created with Python 3.10[Py-3.10] Dynamic MLflow Model Scorerpython-scorer_mlflow_dynamic_310
mlflowMLFlow non-H2O.ai models created with Python 3.11[Py-3.11] Dynamic MLflow Model Scorerpython-scorer_mlflow_dynamic_311
mlflowMLFlow non-H2O.ai models created with Python 3.12[Py-3.12] Dynamic MLflow Model Scorerpython-scorer_mlflow_dynamic_312

MLflow Dynamic Runtime​

The MLflow Dynamic Runtime lets you deploy MLflow models with diverse dependencies in H2O MLOps. The following steps describe how to deploy a dynamic MLflow runtime deployment in H2O MLOps.

note

For an example of how to train a dynamic runtime, see Train a dynamic runtime.

  1. Save your model using the mlflow.pyfunc.save_model function call. Use the pip_requirements parameter to specify the Python package dependencies required by the model.

    mlflow.pyfunc.save_model(
    path=...,
    python_model=...,
    artifacts=...,
    signature=...,
    pip_requirements=..., # <- Use this parameter to override libs for dynamic runtime
    )
  2. After saving the model, create a zip archive of the saved model directory. Ensure that a requirements file (requirements.txt) that lists all dependencies is included in the zip archive. The following is an example of the expected structure for the zip file from a TensorFlow model:

    tf-model-py310
    β”œβ”€β”€ MLmodel
    β”œβ”€β”€ artifacts
    β”‚Β Β  └── tf.h5
    β”œβ”€β”€ conda.yaml
    β”œβ”€β”€ python_env.yaml
    β”œβ”€β”€ python_model.pkl
    └── requirements.txt
  3. Depending on whether you are using Python 3.9 or Python 3.10, select from one of the following options:

    • [PY-3.9] MLflow Dynamic Model Scorer
    • [PY-3.10] MLflow Dynamic Model Scorer
note

The MLflow Dynamic Runtime has a fixed MLflow dependency, which is MLflow 1.26.1. This means that the MLflow Dynamic Runtime is not guaranteed to work with a different version of MLflow model.

Example: Train a dynamic runtime model​

The following example demonstrates how to train a dynamic runtime with TensorFlow:

# Import libraries
import mlflow
import pandas as pd
import shutil
import tensorflow as tf
from sklearn import datasets

# Load and prepare data
diabetes = datasets.load_diabetes()
X = diabetes.data[:, 2:3] # Use only one feature for simplicity
y = diabetes.target

# Build and train TensorFlow model
tf_model = tf.keras.models.Sequential([
tf.keras.layers.Dense(1, input_dim=1)
])
tf_model.compile(optimizer='adam', loss='mean_squared_error')
tf_model.fit(X, y, epochs=10)

tf_model_path = "tf.h5"

tf_model.save(tf_model_path, save_format="h5")


# Enable the TensorFlow model to be used in the Pyfunc format
class PythonTFmodel(mlflow.pyfunc.PythonModel):
def load_context(self, context):
import tensorflow as tf
self.model = tf.keras.models.load_model(context.artifacts["model"])

def predict(self, context, model_input):
tf_out = self.model.predict(model_input)
return pd.DataFrame(tf_out, columns=["db_progress"])


# Generate signature from your model definition
model = PythonTFmodel()
context = mlflow.pyfunc.PythonModelContext(model_config=dict(), artifacts={"model": tf_model_path})
model.load_context(context)
x = pd.DataFrame(X, columns=["dense_input"])
y = model.predict(context, x)
signature = mlflow.models.signature.infer_signature(x, y)

# Specify a file path where the model will be saved
mlflow_model_path = "./tf-model-py310"

# Save model using MLflow
mlflow.pyfunc.save_model(
path=mlflow_model_path,
python_model=PythonTFmodel(),
signature=signature,
artifacts={"model": tf_model_path},
pip_requirements=["tensorflow"]
)

# Package model as a zip archive
shutil.make_archive(
mlflow_model_path, "zip", mlflow_model_path
)

The following is the structure of the zip file that is generated in the preceding example:

tf-model-py310
β”œβ”€β”€ MLmodel
β”œβ”€β”€ artifacts
β”‚Β Β  └── tf.h5
β”œβ”€β”€ conda.yaml
β”œβ”€β”€ python_env.yaml
β”œβ”€β”€ python_model.pkl
└── requirements.txt

Generic Ephemeral volumes​

The custom additional volumes feature now supports emptyDir volumes and ephemeral volumes.

note

The storageClassName property for volumes is optional. If not provided, the default storage class will be used.

Example configuration​

# Custom additional volumes with selected mount paths.
# This section, as well as each of its fields, is optional.
volume-mounts = [
{
name = "ephemeral_volume"
type = "ephemeral"
properties = [
{ name = "size", value = "1Gi" }
]
paths = ["/ephemeral_volume_1", "/ephemeral_volume_2"]
},
{
name = "emptyDir_volume"
type = "emptyDir"
properties = [
{ name = "medium", value = "Memory" }
]
paths = ["/emptyDir_volume_1", "/emptyDir_volume_2"]
}
]

YAML configuration​

The volumeMounts section should be added to the runtime specification of the Helm Chart.

runtimes:
volumeMounts:
- name: "dev-shm"
type: "ephemeral"
properties:
size: "1Gi"
paths: ["/tmp"]

H2O Hydrogen Torch runtime​

Send request to HT text based model​

payload = deployment_sample_request
payload["rows"] = [[f"this is a test for row {i}"] for i in range(10)]

r = requests.post(score_url, json=payload)

Send request to HT text span based model​

payload = deployment_sample_request
payload["fields"] = ["question", "context"]
payload["rows"] = [[f"this is a test for question {i}", f"this is a test for context {i}"] for i in range(10)]

r = requests.post(score_url, json=payload)

Send request to HT audio based model​

def read_binary(file_path):
return open(file_path, 'rb')

files = [
('files', (f'test_audio_{i}.ogg', read_binary(hydrogen_torch_test_audio_file), 'application/octet-stream'))
for i in range(10)
]
metadata = (
'scoreMediaRequest',
(
None,
json.dumps({
"fields": ["input"],
"media_fields": ["input"],
"rows": [[f"test_audio_{i}.ogg"] for i in range(10)]
}),
"application/json"
)
)
files.append(metadata)

score_url = score_url.replace('score', 'media-score')
r = requests.post(score_url, files=files)

Send request to HT image based model​

def read_binary(file_path):
return open(file_path, 'rb')

files = [
('files', (f'test_image_{i}.jpg', read_binary(hydrogen_torch_test_image_file), 'image/jpg'))
for i in range(10)
]
metadata = (
'scoreMediaRequest',
(
None,
json.dumps({
"fields": ["input"],
"media_fields": ["input"],
"rows": [[f"test_image_{i}.jpg"] for i in range(10)]
}),
"application/json"
)
)
files.append(metadata)

score_url = score_url.replace('score', 'media-score')
r = requests.post(score_url, files=files)

Feedback