Champion/Challenger deployment
This example demonstrates how you can deploy a Champion/Challenger deployment using Driverless AI. It creates one dataset, two experiments (champion
experiment and challenger
experiment), and one project. The dataset and experiments are then linked to the project and deployed to the DEV
environment in MLOps.
- Install Driverless AI.
- Launch a new instance of Driverless AI.
- You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.
Constant | Value | Description |
---|---|---|
MLOPS_API_URL | Usually: https://api.mlops.my.domain | Defines the URL for the MLOps Gateway component. You can verify the correct URL by navigating to the API URL in your browser. It should provide a page with a list of available routes. |
TOKEN_ENDPOINT_URL | https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token | Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided. |
REFRESH_TOKEN | <your-refresh-token> | Defines the user's refresh token |
CLIENT_ID | <your-client-id> | Sets the client id for authentication. This is the client you will be using to connect to MLOps. |
PROJECT_NAME | ChampionChallengerDeploymentExample | Defines a project name that the script will be using. |
DRIVERLESS_URL | <your-dai-instance-url> | Defines the URL of the Driverless AI instance you have launched. |
TRAIN_FILE_URL | s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv | Defines the example training dataset. |
TARGET_COLUMN | DEFAULT_PAYMENT_NEXT_MONTH | Defines the target column for the experiment. |
DEPLOYMENT_ENVIRONMENT | DEV | Defines the target deployment environment. |
REFRESH_STATUS_INTERVAL | 1.0 | Defines a refresh interval for the deployment health check. |
MAX_WAIT_TIME | 300 | Defines maximum waiting time for the deployment to become healthy. |
The following steps demonstrate how you can use MLOps Python client to deploy a Champion/Challenger deployment.
Change the values of the following constants in your
ChampionChallengerDeploymentExample.py
file as given in the preceding data table.ChampionChallengerDeploymentExample.py### Constants
MLOPS_API_URL = <MLOPS_API_URL>
TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
REFRESH_TOKEN = <REFRESH_TOKEN>
CLIENT_ID = <CLIENT_ID>
PROJECT_NAME = <PROJECT_NAME>
DRIVERLESS_URL = <DRIVERLESS_URL>
TRAIN_FILE_URL = <TRAIN_FILE_URL>
TARGET_COLUMN = <TARGET_COLUMN>
DEPLOYMENT_ENVIRONMENT = <DEPLOYMENT_ENVIRONMENT>
REFRESH_STATUS_INTERVAL = <REFRESH_STATUS_INTERVAL>
MAX_WAIT_TIME = <MAX_WAIT_TIME>ChampionChallengerDeploymentExample.py### Constants
MLOPS_API_URL = "https://api.mlops.my.domain"
TOKEN_ENDPOINT_URL="https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token"
REFRESH_TOKEN="<your-refresh-token>"
CLIENT_ID="<your-mlops-client>"
DRIVERLESS_URL = "<your-dai-instance-url>"
PROJECT_NAME = "ChampionChallengerDeploymentExample"
TRAIN_FILE_URL = "s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv"
TARGET_COLUMN = "DEFAULT_PAYMENT_NEXT_MONTH"
DEPLOYMENT_ENVIRONMENT = "DEV"
REFRESH_STATUS_INTERVAL = 1.0
MAX_WAIT_TIME = 300Run the
ChampionChallengerDeploymentExample.py
file.python3 ChampionChallengerDeploymentExample.py
Complete 100.00% - [4/4] Computed stats for column DEFAULT_PAYMENT_NEXT_MONTH
Experiment launched at: <your-dai-url>/#/experiment?key=8ce19b1a-13c0-11ed-9527-52c341832785
Complete 100.00% - Status: Complete
Experiment launched at: <your-dai-url>/#/experiment?key=cbb1a4b6-13c0-11ed-9527-52c341832785
Complete 100.00% - Status: Complete
Deployment has become healthyNavigate to the launched Driverless AI instance from your browser and click the Experiments tab. The script has created two experiments,
cc_experiment_champion
andcc_experiment_challenger
.Click the experiment name
cc_experiment_champion
to view details of the champion experiment.Click the experiment name
cc_experiment_challenger
to view details of the challenger experiment.NoteFor more information about experiments in Driverless AI, see Driverless AI documentation.
Click the Projects tab in the Driverless AI instance and select the project
Champion/ChallengerDeploymentExample
to view the created project.NoteFor more information about projects in Driverless AI, see Driverless AI documentation.
Finally, navigate to MLOps and click the project name
Champion/ChallengerDeploymentExample
under Projects to view the deployments.NoteFor more information about model deployments in MLOps, see Understand deployments in MLOps.
Example walkthrough
The following steps provide a walkthrough of each of the sections in the ChampionChallengerDeploymentExample.py
file.
Include the helper function, which waits for the deployment to be healthy.
Include the following function
run_dai_experiment
, which starts a Driverless AI experiment.ChampionChallengerDeploymentExample.pydef run_dai_experiment(
dai_client: driverlessai.Client,
train_dataset,
experiment_name: str,
accuracy_setting: int,
time_setting: int,
interpretability_setting: int,
task: str = "classification",
scorer: str = "AUC"
):
# Starting a basic credit card experiment with minimal parameters.
exp_cc = dai_client.experiments.create(
name=experiment_name,
train_dataset=train_dataset,
target_column=TARGET_COLUMN,
task=task,
accuracy=accuracy_setting,
time=time_setting,
interpretability=interpretability_setting,
scorer=scorer,
enable_gpus=False,
seed=1234,
cols_to_drop=["ID"],
)
return exp_ccInclude the following function to get the artifact ID from an experiment in order to deploy it.
ChampionChallengerDeploymentExample.pydef get_artifact_id_from_experiment(
mlops_client,
dai_experiment
):
# Looking for the ID of the artifact to deploy.
artifact_list = mlops_client.storage.artifact.list_entity_artifacts(
mlops.StorageListEntityArtifactsRequest(entity_id=dai_experiment.key)
).artifact
for ar in artifact_list:
if ar.type == "dai/mojo_pipeline":
return ar.id
else:
raise LookupError(f"Could not find 'dai/mojo_pipeline' artifact.")Customize the composition of the deployment.
ChampionChallengerDeploymentExample.pydef create_dai_mojo_composition(experiment_id: str, artifact_id: str):
return mlops.DeployDeploymentComposition(
experiment_id=experiment_id,
artifact_id=artifact_id,
deployable_artifact_type_name="dai_mojo_pipeline",
artifact_processor_name="dai_mojo_pipeline_extractor",
runtime_name="dai_mojo_runtime",
)Set up the Driverless AI client.
ChampionChallengerDeploymentExample.pydai_client = driverlessai.Client(
address=DRIVERLESS_URL, token_provider=mlops_token_provider
)Fetch the credit card dataset from S3 as the training dataset and start two experiments
exp_cc_champion
andexp_cc_challenger
.ChampionChallengerDeploymentExample.py# Fetching the credit card dataset from S3.
ds_train = dai_client.datasets.create(data=TRAIN_FILE_URL, data_source="s3")
exp_cc_champion = run_dai_experiment(
dai_client=dai_client,
train_dataset=ds_train,
experiment_name="cc_experiment_champion",
accuracy_setting=1,
time_setting=1,
interpretability_setting=1
)
exp_cc_challenger = run_dai_experiment(
dai_client=dai_client,
train_dataset=ds_train,
experiment_name="cc_experiment_challenger",
accuracy_setting=2,
time_setting=2,
interpretability_setting=2
)Create a project in Driverless AI and link the fetched dataset to the project. Then link both experiments
exp_cc_champion
andexp_cc_challenger
to the project.ChampionChallengerDeploymentExample.py# Creating a project in DAI
prj = dai_client.projects.create(PROJECT_NAME)
# Linking the dataset to the project.
prj.link_dataset(ds_train, dataset_type="train_dataset")
# Linking both experiments to the project
prj.link_experiment(exp_cc_champion)
prj.link_experiment(exp_cc_challenger)
Get the artifact IDs for the experiments.
ChampionChallengerDeploymentExample.pyartifact_id_champion = get_artifact_id_from_experiment(
mlops_client=mlops_client, dai_experiment=exp_cc_champion
)
artifact_id_challenger = get_artifact_id_from_experiment(
mlops_client=mlops_client, dai_experiment=exp_cc_challenger
)Customize the composition of the deployment to perform a Champion/Challenger deployment and specify the deployment as a split deployment.
ChampionChallengerDeploymentExample.py# Customize the composition of the deployment to perform Champion/Challenger deployment
champion_element = mlops.DeployShadowElement(
deployment_composition=create_dai_mojo_composition(
experiment_id=exp_cc_champion.key, artifact_id=artifact_id_champion
),
)
challenger_element = mlops.DeployShadowElement(
deployment_composition=create_dai_mojo_composition(
experiment_id=exp_cc_challenger.key, artifact_id=artifact_id_challenger
),
)
# Specify the deployment as a split deployment.
to_deploy = mlops.DeployDeployment(
project_id=prj.key,
deployment_environment_id=deployment_env_id,
shadow_deployment=mlops.DeployShadowDeployment(
primary_element=champion_element, secondary_element=challenger_element
),
)Finally, create a deployment and wait for the deployment to become healthy. This analyzes and sets the metadata and parameters of the model, and deploys it to the
DEV
environment.
- Submit and view feedback for this page
- Send feedback about H2O MLOps to cloud-feedback@h2o.ai