Skip to main content
Version: v0.65.1

Champion/Challenger deployment

This example demonstrates how you can deploy a Champion/Challenger deployment using Driverless AI. It creates one dataset, two experiments (champion experiment and challenger experiment), and one project. The dataset and experiments are then linked to the project and deployed to the DEV environment in MLOps.

Before you begin
  • Install Driverless AI.
  • Launch a new instance of Driverless AI.
  • You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.
ConstantValueDescription
MLOPS_API_URLUsually: https://api.mlops.my.domainDefines the URL for the MLOps Gateway component. You can verify the correct URL by navigating to the API URL in your browser. It should provide a page with a list of available routes.
TOKEN_ENDPOINT_URL
https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token
Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided.
REFRESH_TOKEN<your-refresh-token>Defines the user's refresh token
CLIENT_ID<your-client-id>Sets the client id for authentication. This is the client you will be using to connect to MLOps.
PROJECT_NAMEChampionChallengerDeploymentExampleDefines a project name that the script will be using.
DRIVERLESS_URL<your-dai-instance-url>Defines the URL of the Driverless AI instance you have launched.
TRAIN_FILE_URL
s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv
Defines the example training dataset.
TARGET_COLUMNDEFAULT_PAYMENT_NEXT_MONTHDefines the target column for the experiment.
DEPLOYMENT_ENVIRONMENTDEVDefines the target deployment environment.
REFRESH_STATUS_INTERVAL1.0Defines a refresh interval for the deployment health check.
MAX_WAIT_TIME300Defines maximum waiting time for the deployment to become healthy.

The following steps demonstrate how you can use MLOps Python client to deploy a Champion/Challenger deployment.

  1. Download the ChampionChallengerDeploymentExample.py file.

  2. Change the values of the following constants in your ChampionChallengerDeploymentExample.py file as given in the preceding data table.

    ChampionChallengerDeploymentExample.py
    ### Constants
    MLOPS_API_URL = <MLOPS_API_URL>
    TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
    REFRESH_TOKEN = <REFRESH_TOKEN>
    CLIENT_ID = <CLIENT_ID>
    PROJECT_NAME = <PROJECT_NAME>
    DRIVERLESS_URL = <DRIVERLESS_URL>
    TRAIN_FILE_URL = <TRAIN_FILE_URL>
    TARGET_COLUMN = <TARGET_COLUMN>
    DEPLOYMENT_ENVIRONMENT = <DEPLOYMENT_ENVIRONMENT>
    REFRESH_STATUS_INTERVAL = <REFRESH_STATUS_INTERVAL>
    MAX_WAIT_TIME = <MAX_WAIT_TIME>
    ChampionChallengerDeploymentExample.py
    ### Constants
    MLOPS_API_URL = "https://api.mlops.my.domain"
    TOKEN_ENDPOINT_URL="https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token"
    REFRESH_TOKEN="<your-refresh-token>"
    CLIENT_ID="<your-mlops-client>"
    DRIVERLESS_URL = "<your-dai-instance-url>"
    PROJECT_NAME = "ChampionChallengerDeploymentExample"
    TRAIN_FILE_URL = "s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv"
    TARGET_COLUMN = "DEFAULT_PAYMENT_NEXT_MONTH"
    DEPLOYMENT_ENVIRONMENT = "DEV"
    REFRESH_STATUS_INTERVAL = 1.0
    MAX_WAIT_TIME = 300
  3. Run the ChampionChallengerDeploymentExample.py file.

    python3 ChampionChallengerDeploymentExample.py
    Complete 100.00% - [4/4] Computed stats for column DEFAULT_PAYMENT_NEXT_MONTH
    Experiment launched at: <your-dai-url>/#/experiment?key=8ce19b1a-13c0-11ed-9527-52c341832785
    Complete 100.00% - Status: Complete
    Experiment launched at: <your-dai-url>/#/experiment?key=cbb1a4b6-13c0-11ed-9527-52c341832785
    Complete 100.00% - Status: Complete
    Deployment has become healthy
  4. Navigate to the launched Driverless AI instance from your browser and click the Experiments tab. The script has created two experiments, cc_experiment_champion and cc_experiment_challenger.

    Champion/Challenger deployment experiments

    Click the experiment name cc_experiment_champion to view details of the champion experiment.

    Champion/Challenger deployment experiment champion

    Click the experiment name cc_experiment_challenger to view details of the challenger experiment.

    Champion/Challenger deployment experiment challenger

    Note

    For more information about experiments in Driverless AI, see Driverless AI documentation.

  1. Click the Projects tab in the Driverless AI instance and select the project Champion/ChallengerDeploymentExample to view the created project.

    Champion/Challenger deployment project

    Note

    For more information about projects in Driverless AI, see Driverless AI documentation.

  2. Finally, navigate to MLOps and click the project name Champion/ChallengerDeploymentExample under Projects to view the deployments.

    Champion/Challenger deployment models

    Note

    For more information about model deployments in MLOps, see Understand deployments in MLOps.

Example walkthrough

The following steps provide a walkthrough of each of the sections in the ChampionChallengerDeploymentExample.py file.

  1. Include the helper function, which waits for the deployment to be healthy.

  2. Include the following function run_dai_experiment, which starts a Driverless AI experiment.

    ChampionChallengerDeploymentExample.py
    def run_dai_experiment(
    dai_client: driverlessai.Client,
    train_dataset,
    experiment_name: str,
    accuracy_setting: int,
    time_setting: int,
    interpretability_setting: int,
    task: str = "classification",
    scorer: str = "AUC"
    ):
    # Starting a basic credit card experiment with minimal parameters.
    exp_cc = dai_client.experiments.create(
    name=experiment_name,
    train_dataset=train_dataset,
    target_column=TARGET_COLUMN,
    task=task,
    accuracy=accuracy_setting,
    time=time_setting,
    interpretability=interpretability_setting,
    scorer=scorer,
    enable_gpus=False,
    seed=1234,
    cols_to_drop=["ID"],
    )

    return exp_cc
  3. Include the following function to get the artifact ID from an experiment in order to deploy it.

    ChampionChallengerDeploymentExample.py
    def get_artifact_id_from_experiment(
    mlops_client,
    dai_experiment
    ):
    # Looking for the ID of the artifact to deploy.
    artifact_list = mlops_client.storage.artifact.list_entity_artifacts(
    mlops.StorageListEntityArtifactsRequest(entity_id=dai_experiment.key)
    ).artifact

    for ar in artifact_list:
    if ar.type == "dai/mojo_pipeline":
    return ar.id
    else:
    raise LookupError(f"Could not find 'dai/mojo_pipeline' artifact.")
  4. Customize the composition of the deployment.

    ChampionChallengerDeploymentExample.py
    def create_dai_mojo_composition(experiment_id: str, artifact_id: str):
    return mlops.DeployDeploymentComposition(
    experiment_id=experiment_id,
    artifact_id=artifact_id,
    deployable_artifact_type_name="dai_mojo_pipeline",
    artifact_processor_name="dai_mojo_pipeline_extractor",
    runtime_name="dai_mojo_runtime",
    )
  5. Set up the token provider using an existing refresh token.

  6. Set up the Driverless AI client.

    ChampionChallengerDeploymentExample.py
    dai_client = driverlessai.Client(
    address=DRIVERLESS_URL, token_provider=mlops_token_provider
    )
  7. Set up the MLOps client.

  8. Fetch the credit card dataset from S3 as the training dataset and start two experiments exp_cc_champion and exp_cc_challenger.

    ChampionChallengerDeploymentExample.py
    # Fetching the credit card dataset from S3.
    ds_train = dai_client.datasets.create(data=TRAIN_FILE_URL, data_source="s3")

    exp_cc_champion = run_dai_experiment(
    dai_client=dai_client,
    train_dataset=ds_train,
    experiment_name="cc_experiment_champion",
    accuracy_setting=1,
    time_setting=1,
    interpretability_setting=1
    )

    exp_cc_challenger = run_dai_experiment(
    dai_client=dai_client,
    train_dataset=ds_train,
    experiment_name="cc_experiment_challenger",
    accuracy_setting=2,
    time_setting=2,
    interpretability_setting=2
    )
  9. Create a project in Driverless AI and link the fetched dataset to the project. Then link both experiments exp_cc_champion and exp_cc_challenger to the project.

    ChampionChallengerDeploymentExample.py
    # Creating a project in DAI
    prj = dai_client.projects.create(PROJECT_NAME)

    # Linking the dataset to the project.
    prj.link_dataset(ds_train, dataset_type="train_dataset")

    # Linking both experiments to the project
    prj.link_experiment(exp_cc_champion)
    prj.link_experiment(exp_cc_challenger)
  1. Fetch the available deployment environments and look up the ID of the selected deployment environment.

  2. Get the artifact IDs for the experiments.

    ChampionChallengerDeploymentExample.py
    artifact_id_champion = get_artifact_id_from_experiment(
    mlops_client=mlops_client, dai_experiment=exp_cc_champion
    )

    artifact_id_challenger = get_artifact_id_from_experiment(
    mlops_client=mlops_client, dai_experiment=exp_cc_challenger
    )
  3. Customize the composition of the deployment to perform a Champion/Challenger deployment and specify the deployment as a split deployment.

    ChampionChallengerDeploymentExample.py
    # Customize the composition of the deployment to perform Champion/Challenger deployment
    champion_element = mlops.DeployShadowElement(
    deployment_composition=create_dai_mojo_composition(
    experiment_id=exp_cc_champion.key, artifact_id=artifact_id_champion
    ),
    )

    challenger_element = mlops.DeployShadowElement(
    deployment_composition=create_dai_mojo_composition(
    experiment_id=exp_cc_challenger.key, artifact_id=artifact_id_challenger
    ),
    )

    # Specify the deployment as a split deployment.
    to_deploy = mlops.DeployDeployment(
    project_id=prj.key,
    deployment_environment_id=deployment_env_id,
    shadow_deployment=mlops.DeployShadowDeployment(
    primary_element=champion_element, secondary_element=challenger_element
    ),
    )
  4. Finally, create a deployment and wait for the deployment to become healthy. This analyzes and sets the metadata and parameters of the model, and deploys it to the DEV environment.


Feedback