Deployment scorer

You can use the H2O MLOps Python client to score against deployments. This page explains how to retrieve deployment scorer, access the scorer's endpoints, and send scoring requests using various capabilities such as prediction intervals, Shapley values, and media inputs.

Prerequisites

Before you begin,

1. Import the necessary Python packages. For instructions, see Step 1: Import the required packages.
2. Connect to H2O MLOps. For instructions, see Connect to H2O MLOps.
3. Create a workspace. For instructions, see Create a workspace.
4. Create one or two experiments. For instructions, see Create an experiment.
5. Create models and register the experiments with them. For instructions, see Register an experiment with a model.
6. Create a deployment. For instructions, see Create a deployment.

View deployment scorers

List deployment scorers

List all deployment scorers in a workspace:

Input:

workspace.deployments.scorers()

Output:

   | uid                                  | scoring_endpoint
---+--------------------------------------+-------------------------------------------------------------------------------------------
 0 | 822fcf3d-6d4c-4948-a71b-2d0b46db82a9 | https://model.dev.mlops-internal.h2o.dev/822fcf3d-6d4c-4948-a71b-2d0b46db82a9/model/score

Filter deployment scorers

Use the scorers() method with key-value arguments to filter the deployment scorers.

Input:

workspace.deployments.scorers(uid="822fcf3d-6d4c-4948-a71b-2d0b46db82a9")

This returns a list of matching deployment scorers as a table.

Output:

   | uid                                  | scoring_endpoint
---+--------------------------------------+-------------------------------------------------------------------------------------------
 0 | 822fcf3d-6d4c-4948-a71b-2d0b46db82a9 | https://model.dev.mlops-internal.h2o.dev/822fcf3d-6d4c-4948-a71b-2d0b46db82a9/model/score

Retrieve a deployment scorer

To retrieve the scorer object for a deployment:

Input:

scorer = deployment.scorer
scorer

note

You can also retrieve a deployment scorer from the list returned by scorers() using indexing.
For example, scorer = workspace.deployments.scorers(key=value)[index]. The key and value arguments are optional.

Output:

<class 'h2o_mlops._deployments.MLOpsDeploymentScorer(
   api_base_url='https://model.dev.mlops-internal.h2o.dev/822fcf3d-6d4c-4948-a71b-2d0b46db82a9',
   capabilities_endpoint='https://model.dev.mlops-internal.h2o.dev/822fcf3d-6d4c-4948-a71b-2d0b46db82a9/model/capabilities',
   schema_endpoint='https://model.dev.mlops-internal.h2o.dev/822fcf3d-6d4c-4948-a71b-2d0b46db82a9/model/schema',
   sample_request_endpoint='https://model.dev.mlops-internal.h2o.dev/822fcf3d-6d4c-4948-a71b-2d0b46db82a9/model/sample_request',
   scoring_endpoint='https://model.dev.mlops-internal.h2o.dev/822fcf3d-6d4c-4948-a71b-2d0b46db82a9/model/score',
)'>

Deployment scorer properties

A deployment scorer has the following main properties:

• api_base_url: The base URL for all REST API interactions.
• readyz_endpoint: Endpoint for checking if the scorer is ready to receive requests.
• capabilities_endpoint: Endpoint that lists supported scorer capabilities. For more information, see View scorer capabilities.
• schema_endpoint: Endpoint that returns the input and output schema, including field names and types.
• sample_request_endpoint: Endpoint that returns a sample request payload with placeholder values.
• scoring_endpoint: Endpoint for submitting payloads to get model predictions.
• media_scoring_endpoint: Endpoint for scoring media inputs such as images, audio, or text.
• contributions_endpoint: Endpoint that returns feature contributions (Shapley values), if supported.

Access endpoints

You can use scorer methods to interact with deployment endpoints. Each method that sends a request to an endpoint accepts the following optional parameters:

• auth_value: Optional[str] = None – The deployment authorization value, such as a passphrase or access token. Required only if the endpoint is secured. For more information on security options, see Security options.
• timeout: Optional[float] = 5 – The timeout in seconds for the HTTP request.

note

For the OIDC_AUTH security option, use a valid access token as the authentication value. For example:

import h2o_authn

token_provider = h2o_authn.TokenProvider(
   refresh_token=...,
   client_id=...,
   token_endpoint_url=...,
)
auth_value = token_provider()

H2O MLOps client automatically tries to fetch the access token when none is provided.

View scorer state

To view the scorer’s state:

Input:

scorer.state()

Output:

'Ready'

Check if the scorer is ready

To check if the scorer is ready to receive requests:

Input:

scorer.is_ready()

Output:

True

View scorer capabilities

To view the scorer’s supported capabilities:

Input:

scorer.capabilities()

Output:

['SCORE_PREDICTION_INTERVAL', 'SCORE']

The available scorer capabilities are:

• SCORE: Enables standard predictions for structured data.
• SCORE_PREDICTION_INTERVAL: Allows returning prediction intervals.
• CONTRIBUTION_ORIGINAL: Supports computing Shapley values for original input features.
• CONTRIBUTION_TRANSFORMED: Supports computing Shapley values for transformed features.
• MEDIA: Accepts media inputs such as images, audio, or text for scoring.
• TEST_TIME_AUGMENTATION: Eligible to apply test-time augmentation and aggregate results.

For more information, see Advanced capabilities.

View schema

To view the input and output schema, including column names and their data types:

Input:

scorer.schema()

Output:

{
'id': '92aa5dee-5b6b-4a28-a19f-0156ae0bde34',
'schema': {
   'inputFields': [
     {'name': 'Origin', 'dataType': 'Str'},
     {'name': 'Dest', 'dataType': 'Str'},
     {'name': 'fDayofMonth', 'dataType': 'Str'},
     {'name': 'fYear', 'dataType': 'Str'},
     {'name': 'UniqueCarrier', 'dataType': 'Str'},
     {'name': 'fDayOfWeek', 'dataType': 'Str'},
     {'name': 'fMonth', 'dataType': 'Str'},
     {'name': 'IsDepDelayed', 'dataType': 'Str'}],
   'outputFields': [
     {'name': 'Distance', 'dataType': 'Float64'}]}
}

note

The id in the response identifies the experiment associated with the scorer. This is especially helpful in multi-model deployments.

Example: View the experiments associated with the deployment.

Input:

deployment.experiments

Output:

   | name                 | uid
---+----------------------+--------------------------------------
 0 | H2O3 MOJO experiment | 92aa5dee-5b6b-4a28-a19f-0156ae0bde34

The experiment ID matches the one returned by the scorer.

Generate a sample request

To generate a sample scoring payload with placeholder values:

Input:

scorer.sample_request()

Output:

{'fields': ['Origin',
 'Dest',
 'fDayofMonth',
 'fYear',
 'UniqueCarrier',
 'fDayOfWeek',
 'fMonth',
 'IsDepDelayed'],
'rows': [['text', 'text', 'text', 'text', 'text', 'text', 'text', 'text']]}

Create a payload

Here’s an example of a manually defined payload with multiple rows:

payload = {
   "fields": [
       "Origin", "Dest", "fDayofMonth", "fYear", "UniqueCarrier", "fDayOfWeek", "fMonth", "IsDepDelayed",
   ],
   "rows": [
       ["text", "text", "text", "text", "text", "text", "text", "text"],
       ["text", "text", "text", "text", "text", "text", "text", "text"],
       ["text", "text", "text", "text", "text", "text", "text", "text"],
   ]
}

Score against the deployment

If the scorer supports the SCORE capability, you can send a payload to score the model:

Input:

scorer.score(payload=payload)

Output:

{'id': '92aa5dee-5b6b-4a28-a19f-0156ae0bde34',
'fields': ['Distance'],
'score': [['713.7770420135266'],
 ['713.7770420135266'],
 ['713.7770420135266']]}

note

The id in the response identifies the experiment associated with the scorer. This is especially helpful in multi-model deployments.

Advanced capabilities

Prediction intervals

If the scorer supports the SCORE_PREDICTION_INTERVAL capability, you can request prediction intervals by adding the following flag to the payload:

payload["requestPredictionIntervals"] = True

Shapley values

If the scorer supports the CONTRIBUTION_ORIGINAL or CONTRIBUTION_TRANSFORMED capability, you can request feature contributions by setting the requestShapleyValueType field in the payload. If the scorer doesn’t support the requested capability, it returns an error.

Example:

payload["requestShapleyValueType"] = "ORIGINAL"

• ORIGINAL: Returns Shapley values for the original input features.
• TRANSFORMED: Returns Shapley values for the transformed features.

The scorer returns an array of rows containing the requested Shapley values, aligned with the fields specified in the request.

note

If you only want feature contributions and not prediction results, use the score_contributions() method instead of score(). The input arguments are the same.

Media scoring

You can score media inputs, such as text, images, or audio, using either the score() or score_media() method.

Option 1: Use score() with base64-encoded content

The score() method expects all media content as base64-encoded strings.

import base64

with open("/abs/path/to/file", "rb") as f:
 encoded = base64.b64encode(f.read()).decode()

payload["rows"] = [[encoded]]

Option 2: Use score_media() with file paths

The score_media() method allows you to send raw media files directly by specifying the file paths.

payload["media_fields"] = ["file"]  # This must be a subset of payload["fields"]
payload["rows"] = [["file_name1"], ["file_name2"], ["file_name3"]]

scorer.score_media(
    payload=payload,
    file_paths=[
        "/abs/path/to/file1",
        "/abs/path/to/file2",
        "/abs/path/to/file3",
    ],
)

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O MLOps to cloud-feedback@h2o.ai