Skip to main content
Version: v1.2.0

External deployments

An external deployment enables H2O MLOps to monitor models deployed outside the H2O MLOps platform. Instead of serving a model, the deployment runs a standalone mlops-proxy that captures scoring data and forwards it to the existing H2O MLOps monitoring pipeline. Use it to track a model you have already deployed elsewhere, or to monitor a model that runs in an environment H2O MLOps can't reach.

An external deployment runs in one of two modes:

  • Proxy URL: The proxy forwards each scoring request to a reachable upstream model, returns the upstream response to the caller, and records the request and response pair for monitoring.
  • Recorder: The proxy records pre-paired request and response records that you post to it. It makes no upstream call.

An external deployment has no model, experiment, or composition. You define its monitoring schema directly with MonitoringOptions columns, and an immutable ApiTemplate tells the proxy how to extract those columns from the request and response bodies.

note

You create and manage external deployments with the H2O MLOps Python client.

Prerequisites​

Before you begin, complete the following steps:

  1. Import the necessary Python packages. For instructions, see Step 1: Import the required packages.
  2. Connect to H2O MLOps. For instructions, see Connect to H2O MLOps.
  3. Create a workspace. For instructions, see Create a workspace.

The examples on this page assume that you have assigned the workspace to the variable workspace. Because an external deployment has no model, experiment, or composition, you don't need to create any of those first.

Import the classes used throughout this page:

from h2o_mlops.options import (
   ApiTemplate,
   Column,
   ExternalDeploymentOptions,
   MonitoringOptions,
   RowTemplate,
   SecurityOptions,
   UpstreamAuth,
)
from h2o_mlops.types import (
   ColumnLogicalType,
   DeploymentModeType,
   ExternalDeploymentMode,
   SecurityType,
)

Define the monitoring schema​

Because an external deployment has no experiment, you define the monitored columns directly with MonitoringOptions, and you map each column to a location in the request or response body with an ApiTemplate. Both modes share this setup.

Define the monitored columns​

Use MonitoringOptions to list the input and output columns to monitor:

monitoring_options = MonitoringOptions(
   enabled=True,
   input_columns=[
       Column(name="LIMIT_BAL", logical_type=ColumnLogicalType.NUMERICAL),
       Column(name="AGE", logical_type=ColumnLogicalType.NUMERICAL),
       Column(name="SEX", logical_type=ColumnLogicalType.CATEGORICAL),
   ],
   output_columns=[
       Column(
           name="PREDICTION",
           logical_type=ColumnLogicalType.NUMERICAL,
       ),
   ],
)

Map the columns with an ApiTemplate​

The ApiTemplate tells the proxy how to find each monitored column in the request and response bodies. It has a request template for input columns and a response template for output columns. Each template is a RowTemplate with the following fields:

  • rows_path: A JMESPath expression that yields the array of row objects in the body. For example, "instances" selects the instances array, and "@" selects the whole body.
  • column_paths: A map of each monitored column name to a JMESPath expression, relative to each row object, that locates the column value.
api_template = ApiTemplate(
   request=RowTemplate(
       rows_path="instances",
       column_paths={"LIMIT_BAL": "LIMIT_BAL", "AGE": "AGE", "SEX": "SEX"},
   ),
   response=RowTemplate(
       rows_path="predictions",
       column_paths={"PREDICTION": "score"},
   ),
)
note
  • JMESPath is case-sensitive. The keys in column_paths must match the column names in monitoring_options, and the paths must match the keys in the request and response bodies exactly.
  • The api_template is immutable after you create the deployment. You can't change it with update().
  • You can omit the request or response template when the deployment monitors no columns of that kind.

Provide baseline data​

An external deployment has no experiment or training dataset, so H2O MLOps can't compute monitoring baselines automatically. Provide filled baselines through the baseline_data field of MonitoringOptions, using BaselineData with a NumericalAggregate, CategoricalAggregate, or TextAggregate for each column. The automatic prepare_monitoring_options_from_data_frame helper doesn't apply to external deployments.

For the structure of BaselineData and its aggregates, see Manual configuration.

Proxy URL mode​

Use proxy URL mode when you have already deployed a model outside H2O MLOps and want to monitor it. You point the proxy at your existing endpoint with upstream_url. The proxy forwards each scoring request to that endpoint, adds the configured upstream authentication, returns the upstream response to the caller, and records the request and response pair for monitoring.

Authenticate to the upstream​

UpstreamAuth configures how the proxy authenticates to your upstream model. It requires exactly one of the following fields.

Bearer token​

Store the token in H2O Secure Store and reference it by key:

secret = workspace.secrets.create(name="upstream-token", value=b"<token>")
upstream_auth = UpstreamAuth(bearer_secret_ref=secret.key)

mTLS​

Reference a kubernetes.io/tls Secret that contains tls.crt and tls.key in the proxy namespace:

upstream_auth = UpstreamAuth(mtls_kubernetes_secret="my-upstream-client-cert")

Create the deployment​

Configure the deployment with ExternalDeploymentOptions, then create it with mode=DeploymentModeType.EXTERNAL. Proxy URL mode requires upstream_url.

proxy_options = ExternalDeploymentOptions(
   mode=ExternalDeploymentMode.PROXY_URL,
   api_template=api_template,
   upstream_url="https://example-deployment.external.com/score",
   upstream_auth=upstream_auth,
   readyz_url="https://example-deployment.external.com/readyz",
)

proxy_deployment = workspace.deployments.create(
   name="external-proxy-demo",
   composition_options=None,
   security_options=SecurityOptions(security_type=SecurityType.DISABLED),
   mode=DeploymentModeType.EXTERNAL,
   external_options=proxy_options,
   monitoring_options=monitoring_options,
)
note

For an external deployment, composition_options must be None. The kubernetes_options, vpa_options, pdb_options, and environment_variables arguments don't apply. Set the proxy's scaling and placement through external_options instead. For more information, see Scale and place the proxy.

Score against the deployment​

Post a scoring request to the scoring endpoint. The proxy forwards it to upstream_url, returns the upstream response, and records the pair.

import requests

score_url = proxy_deployment.scorer.scoring_endpoint
payload = {"instances": [{"LIMIT_BAL": 30000, "AGE": 23, "SEX": 2}]}

response = requests.post(score_url, json=payload)
print(response.status_code, response.json())   # 200 -> {"predictions": [{"score": 0.42}]}

The request and response bodies use the upstream model's own shape, not the standard H2O MLOps {"fields": [...], "rows": [...]} tabular format. The body keys must match the api_template column_paths exactly, because JMESPath is case-sensitive. If the deployment uses token security, pass the credential as usual, such as in an Authorization header.

Recorder mode​

Use recorder mode when the model runs in an environment that can't reach H2O MLOps, such as an air-gapped factory floor. In this mode, the proxy makes no upstream call. Instead, you post pre-paired request and response records to the record endpoint, and the proxy captures them for monitoring.

Create the deployment​

Recorder mode uses only mode and api_template:

recorder_options = ExternalDeploymentOptions(
   mode=ExternalDeploymentMode.RECORDER,
   api_template=api_template,
)

recorder_deployment = workspace.deployments.create(
   name="external-recorder-demo",
   composition_options=None,
   security_options=SecurityOptions(security_type=SecurityType.DISABLED),
   mode=DeploymentModeType.EXTERNAL,
   external_options=recorder_options,
   monitoring_options=monitoring_options,
)
note

Recorder mode must not set upstream_url, upstream_auth, or readyz_url. Setting any of them raises a ValueError.

Record a request and response pair​

Post a record that pairs a request body with its response body. The recorder returns 202 Accepted.

import requests

record_url = recorder_deployment.scorer.scoring_endpoint
record = {
   "request": {"instances": [{"LIMIT_BAL": 30000, "AGE": 23, "SEX": 2}]},
   "response": {"predictions": [{"score": 0.42}]},
   "request-timestamp": "2026-06-22T09:39:21.091631769Z",    # optional
   "response-timestamp": "2026-06-22T09:39:21.091631769Z",   # optional
}

response = requests.post(record_url, json=record)
print(response.status_code)   # 202

A record can optionally include request-timestamp and response-timestamp fields (RFC 3339 or ISO 8601, in UTC with a trailing Z) that capture when the request and response actually occurred. This is useful because the recorder can receive a record long after the producer generated the data. If you omit the timestamps, the recorder timestamps the record when it receives it. The body keys must match the api_template column_paths exactly, so that the captured columns line up with monitoring_options.

Record offline traffic on a schedule​

Where the model runs with no access to H2O MLOps, separate capture from shipping:

  1. On the air-gapped machine, append each captured request and response pair to a file as JSON Lines, with one record per line.
  2. On a host that can reach H2O MLOps, run a scheduled job, such as a cron job, that reads the file and posts each record to the record endpoint.

The offline producer writes records to a file:

import json

record = {
   "request": {"instances": [{"LIMIT_BAL": 30000, "AGE": 23, "SEX": 2}]},
   "response": {"predictions": [{"score": 0.42}]},
   # optional -- when the original request and response actually happened:
   "request-timestamp": "2026-06-22T09:39:21.091631769Z",
   "response-timestamp": "2026-06-22T09:39:21.091631769Z",
}

with open("records.jsonl", "a") as f:
   f.write(json.dumps(record) + "\n")

The scheduled shipper reads the file and records each pair:

import json
import requests

record_url = "https://mlops.example.com/<deployment-id>/model/record"

with open("records.jsonl") as f:
   for line in f:
       line = line.strip()
       if not line:
           continue
       requests.post(record_url, json=json.loads(line))   # returns 202 Accepted

Get the scoring or record URL and the schema​

Both modes expose a scorer. Use it to discover the endpoint URLs and the monitored schema:

scorer = deployment.scorer

scorer.api_base_url       # https://mlops.example.com/<deployment-id>
scorer.scoring_endpoint   # proxy_url -> .../model/score; recorder -> .../model/record
scorer.schema_endpoint    # .../model/schema
scorer.schema()           # {"input_fields": [...], "output_fields": [...]}
scorer.sample_request()   # an example record, including request, response, and timestamps

For a recorder, sample_request() returns the full record shape, including the optional timestamp fields:

Input:

recorder_deployment.scorer.sample_request()

Output:

{
   "request": {"instances": [{"AGE": 1.0, "LIMIT_BAL": 1.0, "SEX": "SEX"}]},
   "response": {"predictions": [{"score": 1.0}]},
   "request-timestamp": "2026-06-22T09:39:21.091631769Z",
   "response-timestamp": "2026-06-22T09:39:21.091631769Z",
}

For more information about the scorer and its properties, see Deployment scorer.

Scale and place the proxy​

ExternalDeploymentOptions exposes Kubernetes scaling and placement shortcuts for the proxy pod:

  • replicas: The number of proxy pods. Set 0 to scale to zero. When unset, it defaults to 1.
  • affinity: A Kubernetes affinity shortcut.
  • toleration: A Kubernetes toleration shortcut.
proxy_options = ExternalDeploymentOptions(
   mode=ExternalDeploymentMode.PROXY_URL,
   api_template=api_template,
   upstream_url="https://example-deployment.external.com/score",
   upstream_auth=upstream_auth,
   readyz_url="https://example-deployment.external.com/readyz",
   replicas=2,
   affinity="<affinity-shortcut>",
   toleration="<toleration-shortcut>",
)
note

To check the allowed affinity and toleration values, run mlops.configs.allowed_k8s_affinities and mlops.configs.allowed_k8s_tolerations. For more information, see Node affinity and toleration.

Update an external deployment​

You can update the replicas, affinity, and toleration values, and the proxy URL fields upstream_url, upstream_auth, and readyz_url. The api_template is immutable, so pass the same one you used at creation.

proxy_deployment.update(
   external_options=ExternalDeploymentOptions(
       mode=ExternalDeploymentMode.PROXY_URL,
       api_template=api_template,   # unchanged; immutable
       upstream_url="https://example-deployment.external.com/score",
       upstream_auth=upstream_auth,
       readyz_url="https://example-deployment.external.com/readyz",
       replicas=3,
   )
)

Security, audit, and Kafka​

External deployments support the same platform features as managed deployments:

  • Security: SecurityOptions works the same way, including token authentication. SecurityType.DISABLED is a valid single entry. For more information, see Secure deployments.
  • Audit: Audit logging applies the same way.
  • Kafka: Raw scoring data export to Kafka works for external deployments. For more information, see Raw data export to Kafka.
Feedback