Get metadata for experiments
This example demonstrates how you can get metadata for the experiments in a specified project using the MLOps Python client.
- Create a project in MLOps and link experiments to the project. For this example, we are using the project created in the Driverless AI example.
- Install PrettyTable.
- You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.
Constant | Value | Description |
---|---|---|
MLOPS_API_URL | Usually: https://api.mlops.my.domain | Defines the URL for the MLOps Gateway component. You can verify the correct URL by navigating to the API URL in your browser. It should provide a page with a list of available routes. |
TOKEN_ENDPOINT_URL | https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token | Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided. |
REFRESH_TOKEN | <your-refresh-token> | Defines the user's refresh token |
CLIENT_ID | <your-client-id> | Sets the client id for authentication. This is the client you will be using to connect to MLOps. |
PROJECT_NAME | StandardFlow | Defines a project name that the script will be using. This is the project from which you want to get the experiment metadata. In this example, this is defined as StandardFlow because we are using the project created in the Driverless AI example. |
The following steps demonstrate how you can use the MLOps Python client to get metadata for the experiments in a specified project.
Change the values of the following constants in your
GetExperimentMetadata.py
file as given in the preceding data table.GetExperimentMetadata.py### Constants
MLOPS_API_URL = <MLOPS_API_URL>
TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
REFRESH_TOKEN = <REFRESH_TOKEN>
CLIENT_ID = <CLIENT_ID>
PROJECT_NAME = <PROJECT_NAME>GetExperimentMetadata.py### Constants
MLOPS_API_URL = "https://api.mlops.my.domain"
TOKEN_ENDPOINT_URL="https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token"
REFRESH_TOKEN="<your-refresh-token>"
CLIENT_ID="<your-mlops-client>"
PROJECT_NAME = "StandardFlow"Run the
GetExperimentMetadata.py
file.python3 GetExperimentMetadata.py
This gives the metadata for the experiments in the specified project.
+-----------------------+----------------------------+-------------------+-------------------+-------------------+------------+------------------+-------------+
| Experiment Name | Target Column | Training Duration | Model Size [GB] | DAI Score | DAI Scorer | DAI Status | DAI Version |
+-----------------------+----------------------------+-------------------+-------------------+-------------------+------------+------------------+-------------+
| CreditCardExample-111 | DEFAULT_PAYMENT_NEXT_MONTH | 99s | 1.107462802901864 | 0.778167872321643 | AUC | Status: Complete | 1.10.3.1 |
+-----------------------+----------------------------+-------------------+-------------------+-------------------+------------+------------------+-------------+
Example walkthrough
The following steps provide a walkthrough of each of the sections in the GetExperimentMetadata.py
file.
Get the projects of all users and look for the ID of the specified project.
GetExperimentMetadata.py# Getting all user's projects.
projects: mlops.StorageListProjectsResponse = (
mlops_client.storage.project.list_projects(mlops.StorageListProjectsRequest())
)
# Looking for the ID of the selected project.
for p in projects.project:
if p.display_name == PROJECT_NAME:
prj_id = p.id
break
else:
raise LookupError("Requested project not found")Get a list of all the experiments in the specified project.
GetExperimentMetadata.pyexperiments: mlops.StorageListExperimentsResponse = (
mlops_client.storage.experiment.list_experiments(
mlops.StorageListExperimentsRequest(project_id=prj_id)
)
)Finally, create the output table with the retrieved metadata.
GetExperimentMetadata.py# Setting up output table's field names.
pp = PrettyTable(
field_names=[
"Experiment Name",
"Target Column",
"Training Duration",
"Model Size [GB]",
"DAI Score",
"DAI Scorer",
"DAI Status",
"DAI Version",
]
)
# Creating the output table.
for e in experiments.experiment:
e_data: mlops.StorageExperiment = mlops_client.storage.experiment.get_experiment(
mlops.StorageGetExperimentRequest(
id=e.id,
response_metadata=mlops.StorageKeySelection(pattern=["%"])
# pattern argument is a list of SQL compatible arguments just like for the LIKE statement
# in this case ["%"] means everything
)
).experiment
e_metadata: mlops.StorageMetadata = e_data.metadata
pp.add_row(
[
e_data.display_name,
e_data.parameters.target_column,
e_data.statistics.training_duration,
float(e_metadata.values["dai/model_file_size"].int64_value)
/ (1024 ** 3), # constant to convert from bytes to gigabytes
e_metadata.values["dai/score"].double_value,
e_metadata.values["dai/scorer"].string_value,
e_metadata.values["dai/status/message"].string_value,
e_metadata.values["dai/tool_version"].string_value,
]
)
print(pp)
- Submit and view feedback for this page
- Send feedback about H2O MLOps to cloud-feedback@h2o.ai