Get health metrics

This example demonstrates how you can get the health metrics for deployed model by using the model monitoring service of the MLOps API. This provides metrics such as scoring latency, total predictions, prediction values and predictions, which can be used to evaluate the performance of the model. Currently, only single model deployments are supported by the model monitoring service.

Before you begin

You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.

Constant	Value	Description
MLOPS_API_URL	Usually: https://api.mlops.my.domain	Defines the URL for the MLOps Gateway component.
TOKEN_ENDPOINT_URL	https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token	Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided.
REFRESH_TOKEN	<your-refresh-token>	Defines the user's refresh token
CLIENT_ID	<your-client-id>	Sets the client id for authentication. This is the client you will be using to connect to MLOps.
CLIENT_SECRET	<your-client-secret>	Sets the client secret.
DEPLOYMENT_ID	<your-deployment-id>	Defines a deployment id that the script will be using.

The following steps demonstrate how you can use the MLOps Python client to get the health metrics for a model of a deployment.

1. Download the GetHealthMetrics.py file.

2. Change the values of the following constants in your GetHealthMetrics.py file as given in the preceding data table.

   GetHealthMetrics.py

   ### Constants
   MLOPS_API_URL = <MLOPS_API_URL>
   TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
   REFRESH_TOKEN = <REFRESH_TOKEN>
   CLIENT_ID = <CLIENT_ID>
   CLIENT_SECRET = <CLIENT_SECRET>
   DEPLOYMENT_ID = <DEPLOYMENT_ID>

   GetHealthMetrics.py

   ### Constants
   MLOPS_API_URL = "https://api.mlops.my.domain"
   TOKEN_ENDPOINT_URL="https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token"
   REFRESH_TOKEN="<your-refresh-token>"
   CLIENT_ID="<your-mlops-client>"
   CLIENT_SECRET = "<your-client-secret>"
   DEPLOYMENT_ID = "f9fa4db1-2f30-4b10-ace2-f383a9f74880"   

3. Run the GetHealthMetrics.py file.

   python3 GetHealthMetrics.py

4. This provides the health metrics for the model of the specified deployment.

   Deployment health metrics: {'prediction_values': None,
   'predictions': {'bin': [{'x_max': '2022-07-21T00:00:00Z',
                           'x_min': '2022-07-19T07:28:40Z',
                           'y': '0'},
                           {'x_max': '2022-07-27T00:00:00Z',
                           'x_min': '2022-07-24T00:00:00Z',
                           'y': '0'},
   
                           ....
   
                           {'x_max': '2022-08-17T00:00:00Z',
                           'x_min': '2022-08-14T00:00:00Z',
                           'y': '0'},
                           {'x_max': '2022-08-18T07:28:40Z',
                           'x_min': '2022-08-17T00:00:00Z',
                           'y': '17670'}],
                   'description': '',
                   'name': 'Predictions Over Time',
                   'x_label': 'Time',
                   'y_label': 'Predictions'},
   'scoring_latency': 1.756577830589034,
   'total_predictions': 17670}

Example walkthrough

This section provides a walkthrough of the GetHealthMetrics.py file.

1. Set up the token provider using an existing refresh token and client secret.

2. Set up the MLOps client.

3. List all the monitored deployments of the user by calling the list_monitored_deployments endpoint of the model monitoring service.

   GetHealthMetrics.py

   deployments: mlops.ApiListMonitoredDeploymentsResponse = (
       mlops_client.model_monitoring.monitoring_service.list_monitored_deployments()
   )

4. Select the specified deployment from the list of all monitored deployments by using the defined DEPLOYMENT_ID.

   GetHealthMetrics.py

   for deployment in deployments.deployment:
       if deployment.id == DEPLOYMENT_ID:
           selected_deployment = deployment
           break
   else:
       raise LookupError("Requested project not found")

5. Define the start datetime and end datetime to get the health metrics. Datetime range is defined as 30 days in this example.

   GetHealthMetrics.py

   end_time = datetime.datetime.now(pytz.utc)
   start_time = end_time - datetime.timedelta(days=30)

6. Finally, call the get_model_health_metrics endpoint of the model monitoring service to get the health metrics of the model.

   Note

   A deployment can have multiple models with A/B Test deployment and Champion/ Challenger deployment types. However, only single deployments are currently supported by the model monitoring service. Therefore, the model_id parameter is ignored for now and the API returns the health metrics for the first available model of the deployment.

   GetHealthMetrics.py

   health: mlops.ApiGetModelHealthMetricsResponse = (
       mlops_client.model_monitoring.monitoring_service.get_model_health_metrics(
           deployment_id=selected_deployment.id,
           model_id='xxxx',
           start_date_time=start_time,
           end_date_time=end_time,
       )
   )
   
   print(f"Deployment health metrics: {health}")

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O MLOps to cloud-feedback@h2o.ai