Skip to main content
Version: v0.64.0

Get health metrics

This example demonstrates how you can get the health metrics for deployed model by using the model monitoring service of the MLOps API. This provides metrics such as scoring latency, total predictions, prediction values and predictions, which can be used to evaluate the performance of the model. Currently, only single model deployments are supported by the model monitoring service.

Before you begin

You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.

ConstantValueDescription
MLOPS_API_URLUsually: https://api.mlops.my.domainDefines the URL for the MLOps Gateway component.
TOKEN_ENDPOINT_URL
https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token
Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided.
REFRESH_TOKEN<your-refresh-token>Defines the user's refresh token
CLIENT_ID<your-client-id>Sets the client id for authentication. This is the client you will be using to connect to MLOps.
CLIENT_SECRET<your-client-secret>Sets the client secret.
DEPLOYMENT_ID<your-deployment-id>Defines a deployment id that the script will be using.

The following steps demonstrate how you can use the MLOps Python client to get the health metrics for a model of a deployment.

  1. Download the GetHealthMetrics.py file.

  2. Change the values of the following constants in your GetHealthMetrics.py file as given in the preceding data table.

    GetHealthMetrics.py
    ### Constants
    MLOPS_API_URL = <MLOPS_API_URL>
    TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
    REFRESH_TOKEN = <REFRESH_TOKEN>
    CLIENT_ID = <CLIENT_ID>
    CLIENT_SECRET = <CLIENT_SECRET>
    DEPLOYMENT_ID = <DEPLOYMENT_ID>

  3. Run the GetHealthMetrics.py file.

    python3 GetHealthMetrics.py

  4. This provides the health metrics for the model of the specified deployment.

    Deployment health metrics: {'prediction_values': None,
    'predictions': {'bin': [{'x_max': '2022-07-21T00:00:00Z',
                            'x_min': '2022-07-19T07:28:40Z',
                            'y': '0'},
                            {'x_max': '2022-07-27T00:00:00Z',
                            'x_min': '2022-07-24T00:00:00Z',
                            'y': '0'},

                            ....

                            {'x_max': '2022-08-17T00:00:00Z',
                            'x_min': '2022-08-14T00:00:00Z',
                            'y': '0'},
                            {'x_max': '2022-08-18T07:28:40Z',
                            'x_min': '2022-08-17T00:00:00Z',
                            'y': '17670'}],
                    'description': '',
                    'name': 'Predictions Over Time',
                    'x_label': 'Time',
                    'y_label': 'Predictions'},
    'scoring_latency': 1.756577830589034,
    'total_predictions': 17670}

Example walkthrough​

This section provides a walkthrough of the GetHealthMetrics.py file.

  1. Set up the token provider using an existing refresh token and client secret.

  2. Set up the MLOps client.

  3. List all the monitored deployments of the user by calling the list_monitored_deployments endpoint of the model monitoring service.

    GetHealthMetrics.py
    deployments: mlops.ApiListMonitoredDeploymentsResponse = (
        mlops_client.model_monitoring.monitoring_service.list_monitored_deployments()
    )

  4. Select the specified deployment from the list of all monitored deployments by using the defined DEPLOYMENT_ID.

    GetHealthMetrics.py
    for deployment in deployments.deployment:
        if deployment.id == DEPLOYMENT_ID:
            selected_deployment = deployment
            break
    else:
        raise LookupError("Requested project not found")

  5. Define the start datetime and end datetime to get the health metrics. Datetime range is defined as 30 days in this example.

    GetHealthMetrics.py
    end_time = datetime.datetime.now(pytz.utc)
    start_time = end_time - datetime.timedelta(days=30)

  6. Finally, call the get_model_health_metrics endpoint of the model monitoring service to get the health metrics of the model.

    Note

    A deployment can have multiple models with A/B Test deployment and Champion/ Challenger deployment types. However, only single deployments are currently supported by the model monitoring service. Therefore, the model_id parameter is ignored for now and the API returns the health metrics for the first available model of the deployment.

    GetHealthMetrics.py
    health: mlops.ApiGetModelHealthMetricsResponse = (
        mlops_client.model_monitoring.monitoring_service.get_model_health_metrics(
            deployment_id=selected_deployment.id,
            model_id='xxxx',
            start_date_time=start_time,
            end_date_time=end_time,
        )
    )

    print(f"Deployment health metrics: {health}")
Feedback