Model monitoring

H2O MLOps model monitoring involves observing the performance and behavior of deployed models to ensure they continue to operate effectively and to identify issues such as model drift.

This guide explains how to configure model monitoring during deployment, analyze aggregated data, and identify model drift. Follow the steps below to set up and use model monitoring in H2O MLOps.

Model monitoring with the UI

Step 1: Enable model monitoring

To enable model monitoring for your deployment:

1. In the left navigation panel, click Real-time deployments.
2. Click Create deployment.
   
3. On the Create new deployment page, click Advanced settings.
4. Toggle Enable monitoring to Yes.
   

Step 2: Configure and deploy

During model deployment, configure monitoring to collect and analyze data.

1. If Kafka is available in your environment, provide a pre-created Kafka topic where raw data will be sent.
   For more information, see Raw data export to Kafka.
2. Select the columns you want to monitor.
   
3. Provide baseline data for comparison:
   • Numerical features:
   • Categorical features:
4. Click Deploy.

Step 3: Start scoring

Once the deployment is Healthy, you can begin scoring.

1. In the left navigation panel, click Real-time deployments.
2. Select the deployment you created.
   
3. Go to the Quick scoring tab.
4. Click Score.

Step 4: View aggregated data

To view the scoring aggregates for each monitored column:

1. After scoring completes, go to the Monitoring tab.
2. Wait 3–5 minutes to see data under Scoring aggregates.

Step 5: Analyze data in the monitoring UI

To view and analyze model drift:

1. Click View in monitoring UI.
2. The Superset UI opens.
3. From the SQL drop-down, select SQL Lab.
4. Enter the drift query:
   Example:

   WITH time_series AS (
   -- Get all unique timestamps for the series
   SELECT DISTINCT "timestamp"
   FROM <TABLE_NAME>
   WHERE NOT is_baseline
   AND column_name = <FEATURE_NAME>
   ORDER BY "timestamp"),
   baseline AS (
   -- Get the baseline (expected) data
   SELECT unnest(bin_counts) as count
   FROM <TABLE_NAME>
   WHERE is_baseline
   AND column_name = <FEATURE_NAME>),
   baseline_sum AS (
   -- Calculate sum of baseline counts
   SELECT sum(count) as total
   FROM baseline),
   baseline_props AS (
   -- Calculate baseline proportions
   SELECT count / total        as prop,
   row_number() OVER () as rn
   FROM baseline,
   baseline_sum),
   actual_data AS (
   -- Get actual data for each timestamp
   SELECT "timestamp",
   unnest(bin_counts)                            as count,
   row_number() OVER (PARTITION BY "timestamp") as rn
   FROM <TABLE_NAME>
   WHERE NOT is_baseline
   AND column_name = <FEATURE_NAME>),
   actual_sums AS (
   -- Calculate sums for each timestamp
   SELECT "timestamp",
   sum(count) as total
   FROM actual_data
   GROUP BY "timestamp"),
   actual_props AS (
   -- Calculate proportions for actual data
   SELECT a."timestamp",
   a.count / s.total as prop,
   a.rn
   FROM actual_data a
   JOIN actual_sums s ON a."timestamp" = s."timestamp"
   WHERE s.total >= 0 -- Implementing the Python None return for sum < 200
   ),
   drift_calc AS (
   -- Calculate absolute differences and sum them
   SELECT a."timestamp",
   sum(abs(a.prop - b.prop)) / 2 as drift_score
   FROM actual_props a
   JOIN baseline_props b ON a.rn = b.rn
   GROUP BY a."timestamp")
   -- Final result with timestamps and drift scores
   SELECT "timestamp",
   drift_score
   FROM drift_calc
   ORDER BY "timestamp";
   
5. To save the result as a dataset, go to the Save drop-down and select Save dataset.
   
6. To convert it into a chart and add it to a dashboard:
   1. Go to Charts and customize the chart.
      
   2. Click Save, and add it to a dashboard.
      
   3. Go to Dashboards and select the one you want to view.

You can explore more advanced dashboards for deeper insights.

Configure model monitoring with the Python client

To learn how to configure monitoring for your deployment using the H2O MLOps Python client, see Monitoring setup.

Raw data export to Kafka

Monitoring supports exporting raw scoring data to Kafka. This feature allows users to process scoring data in ways required by their internal regulations. Request and response data from scoring operations can be sent to a specified Kafka topic for downstream processing, auditing, or debugging. This feature is enabled by the MLOPs administrator.

During model deployment with monitoring enabled, scoring data and response data are sent to a default topic configured by the MLOPs administrator. Users can optionally specify a custom Kafka topic where the data are sent for this particular deployment. This allows separating data streams per deployment for improved observability. This configuration has no effect in case the Kafka intgration is disabled by the admonistrator.

note

The custom Kafka topic must exist before deploying the model. Monitoring will not attempt to create the topic automatically.

Once configured, the monitoring captures raw request and response data from scoring operations and forwards it to the configured Kafka topic (global or deployment-specific).

Common use cases for exporting raw data to Kafka include:

• Debugging and inspecting raw scoring payloads
• Auditing input/output for compliance
• Real-time analytics via stream processing systems

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O MLOps to cloud-feedback@h2o.ai