Get metadata for experiments

This example demonstrates how you can get metadata for the experiments in a specified project using the MLOps Python client.

Before you begin

• Create a project in MLOps and link experiments to the project. For this example, we are using the project created in the Driverless AI example.
• Install PrettyTable.
• You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.

Constant	Value	Description
MLOPS_API_URL	Usually: https://api.mlops.my.domain	Defines the URL for the MLOps Gateway component. You can verify the correct URL by navigating to the API URL in your browser. It should provide a page with a list of available routes.
TOKEN_ENDPOINT_URL	https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token	Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided.
REFRESH_TOKEN	<your-refresh-token>	Defines the user's refresh token
CLIENT_ID	<your-client-id>	Sets the client id for authentication. This is the client you will be using to connect to MLOps.
PROJECT_NAME	StandardFlow	Defines a project name that the script will be using. This is the project from which you want to get the experiment metadata. In this example, this is defined as StandardFlow because we are using the project created in the Driverless AI example.

The following steps demonstrate how you can use the MLOps Python client to get metadata for the experiments in a specified project.

1. Download the GetExperimentMetadata.py file.

2. Change the values of the following constants in your GetExperimentMetadata.py file as given in the preceding data table.

   GetExperimentMetadata.py

   ### Constants
   MLOPS_API_URL = <MLOPS_API_URL>
   TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
   REFRESH_TOKEN = <REFRESH_TOKEN>
   CLIENT_ID = <CLIENT_ID>
   PROJECT_NAME = <PROJECT_NAME>

   GetExperimentMetadata.py

   ### Constants
   MLOPS_API_URL = "https://api.mlops.my.domain"
   TOKEN_ENDPOINT_URL="https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token"
   REFRESH_TOKEN="<your-refresh-token>"
   CLIENT_ID="<your-mlops-client>"
   PROJECT_NAME = "StandardFlow"

3. Run the GetExperimentMetadata.py file.

   python3 GetExperimentMetadata.py

4. This gives the metadata for the experiments in the specified project.

   +-----------------------+----------------------------+-------------------+-------------------+-------------------+------------+------------------+-------------+
   |    Experiment Name    |       Target Column        | Training Duration |  Model Size [GB]  |     DAI Score     | DAI Scorer |    DAI Status    | DAI Version |
   +-----------------------+----------------------------+-------------------+-------------------+-------------------+------------+------------------+-------------+
   | CreditCardExample-111 | DEFAULT_PAYMENT_NEXT_MONTH |        99s        | 1.107462802901864 | 0.778167872321643 |    AUC     | Status: Complete |   1.10.3.1  |
   +-----------------------+----------------------------+-------------------+-------------------+-------------------+------------+------------------+-------------+

Example walkthrough

The following steps provide a walkthrough of each of the sections in the GetExperimentMetadata.py file.

1. Set up the token provider using an existing refresh token.

2. Set up the MLOps client.

3. Get the projects of all users and look for the ID of the specified project.

   GetExperimentMetadata.py

   # Getting all user's projects.
   projects: mlops.StorageListProjectsResponse = (
       mlops_client.storage.project.list_projects(mlops.StorageListProjectsRequest())
   )
   
   # Looking for the ID of the selected project.
   for p in projects.project:
       if p.display_name == PROJECT_NAME:
           prj_id = p.id
           break
   else:
       raise LookupError("Requested project not found")

4. Get a list of all the experiments in the specified project.

   GetExperimentMetadata.py

   experiments: mlops.StorageListExperimentsResponse = (
       mlops_client.storage.experiment.list_experiments(
           mlops.StorageListExperimentsRequest(project_id=prj_id)
       )
   )

5. Finally, create the output table with the retrieved metadata.

   GetExperimentMetadata.py

   # Setting up output table's field names.
   pp = PrettyTable(
       field_names=[
           "Experiment Name",
           "Target Column",
           "Training Duration",
           "Model Size [GB]",
           "DAI Score",
           "DAI Scorer",
           "DAI Status",
           "DAI Version",
       ]
   )
   
   # Creating the output table.
   for e in experiments.experiment:
       e_data: mlops.StorageExperiment = mlops_client.storage.experiment.get_experiment(
           mlops.StorageGetExperimentRequest(
               id=e.id,
               response_metadata=mlops.StorageKeySelection(pattern=["%"])
               # pattern argument is a list of SQL compatible arguments just like for the LIKE statement
               # in this case ["%"] means everything
           )
       ).experiment
       e_metadata: mlops.StorageMetadata = e_data.metadata
   
       pp.add_row(
           [
               e_data.display_name,
               e_data.parameters.target_column,
               e_data.statistics.training_duration,
               float(e_metadata.values["dai/model_file_size"].int64_value)
               / (1024 ** 3),  # constant to convert from bytes to gigabytes
               e_metadata.values["dai/score"].double_value,
               e_metadata.values["dai/scorer"].string_value,
               e_metadata.values["dai/status/message"].string_value,
               e_metadata.values["dai/tool_version"].string_value,
           ]
       )
   
   print(pp)

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O MLOps to cloud-feedback@h2o.ai