Version: v0.67.0

A/B Test deployment

This example demonstrates how you can deploy an A/B Test deployment in MLOps using Driverless AI. It creates one dataset, two experiments (experiment A and experiment B), and one project. The dataset and experiments are then linked to the project and deployed to the DEV environment in MLOps.

Before you begin

Install Driverless AI.
Launch a new instance of Driverless AI.
You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.

Constant	Value	Description
`MLOPS_API_URL`	Usually: `https://api.mlops.my.domain`	Defines the URL for the MLOps Gateway component. You can verify the correct URL by navigating to the API URL in your browser. It should provide a page with a list of available routes.
`TOKEN_ENDPOINT_URL`	`https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token`	Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided.
`REFRESH_TOKEN`	`<your-refresh-token>`	Defines the user's refresh token
`CLIENT_ID`	`<your-client-id>`	Sets the client id for authentication. This is the client you will be using to connect to MLOps.
`PROJECT_NAME`	`ABTestDeploymentExample`	Defines a project name that the script will be using.
`DRIVERLESS_URL`	`<your-dai-instance-url>`	Defines the URL of the Driverless AI instance you have launched.
`TRAIN_FILE_URL`	`s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv`	Defines the example training dataset.
`TARGET_COLUMN`	`DEFAULT_PAYMENT_NEXT_MONTH`	Defines the target column for the experiment.
`DEPLOYMENT_ENVIRONMENT`	`DEV`	Defines the target deployment environment.
`REFRESH_STATUS_INTERVAL`	`1.0`	Defines a refresh interval for the deployment health check.
`MAX_WAIT_TIME`	`300`	Defines maximum waiting time for the deployment to become healthy.

The following steps demonstrate how you can use MLOps Python client to deploy an A/B Test deployment.

Download the ABTestDeploymentExample.py file.

Change the values of the following constants in your ABTestDeploymentExample.py file as given in the preceding data table.

ABTestDeploymentExample.py
### Constants
MLOPS_API_URL = <MLOPS_API_URL>
TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
REFRESH_TOKEN = <REFRESH_TOKEN>
CLIENT_ID = <CLIENT_ID>
PROJECT_NAME = <PROJECT_NAME>
DRIVERLESS_URL = <DRIVERLESS_URL>
TRAIN_FILE_URL = <TRAIN_FILE_URL>
TARGET_COLUMN = <TARGET_COLUMN>
DEPLOYMENT_ENVIRONMENT = <DEPLOYMENT_ENVIRONMENT>
REFRESH_STATUS_INTERVAL = <REFRESH_STATUS_INTERVAL>
MAX_WAIT_TIME = <MAX_WAIT_TIME>

ABTestDeploymentExample.py
### Constants
MLOPS_API_URL = "https://api.mlops.my.domain"
TOKEN_ENDPOINT_URL="https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token"
REFRESH_TOKEN="<your-refresh-token>"
CLIENT_ID="<your-mlops-client>"
DRIVERLESS_URL = "<your-dai-instance-url>"
PROJECT_NAME = "ABTestDeploymentExample"
TRAIN_FILE_URL = "s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv"
TARGET_COLUMN = "DEFAULT_PAYMENT_NEXT_MONTH"
DEPLOYMENT_ENVIRONMENT = "DEV"
REFRESH_STATUS_INTERVAL = 1.0
MAX_WAIT_TIME = 300

Run the ABTestDeploymentExample.py file.

python3 ABTestDeploymentExample.py

Complete 100.00% - [4/4] Computed stats for column DEFAULT_PAYMENT_NEXT_MONTH
Experiment launched at: <your-dai-url>/#/experiment?key=a4b3e9f4-12f9-11ed-a7d2-1e7ead84e80b
Complete 100.00% - Status: Complete                                           
Experiment launched at: <your-dai-url>/#/experiment?key=e8e08542-12f9-11ed-a7d2-1e7ead84e80b
Complete 100.00% - Status: Complete                                           
Deployment has become healthy  

Navigate to the launched Driverless AI instance from your browser and click the Experiments tab. The script has created two experiments, cc_experiment_a and cc_experiment_b.
Click the experiment name cc_experiment_a to view details of experiment A.
Click the experiment name cc_experiment_b to view details of experiment B.
Note
For more information about experiments in Driverless AI, see Driverless AI documentation.
Click the Projects tab in the Driverless AI instance and select the project ABTestDeploymentExample to view the created project.
Note
For more information about projects in Driverless AI, see Driverless AI documentation.
Finally, navigate to MLOps and click the project name ABTestDeploymentExample under Projects to view the deployments.
Note
For more information about model deployments in MLOps, see Understand deployments in MLOps.

Example walkthrough

The following steps provide a walkthrough of each of the sections in the ABTestDeploymentExample.py file.

Include the helper function, which waits for the deployment to be healthy.

Include the following function run_dai_experiment, which starts a Driverless AI experiment.

ABTestDeploymentExample.py
def run_dai_experiment(
    dai_client: driverlessai.Client,
    train_dataset,
    experiment_name: str,
    accuracy_setting: int,
    time_setting: int,
    interpretability_setting: int,
    task: str = "classification",
    scorer: str = "AUC"
):
    # Starting a basic credit card experiment with minimal parameters.
    exp_cc = dai_client.experiments.create(
        name=experiment_name,
        train_dataset=train_dataset,
        target_column=TARGET_COLUMN,
        task=task,
        accuracy=accuracy_setting,
        time=time_setting,
        interpretability=interpretability_setting,
        scorer=scorer,
        enable_gpus=False,
        seed=1234,
        cols_to_drop=["ID"],
    )

    return exp_cc

Include the following function to get the artifact ID from an experiment in order to deploy it.

ABTestDeploymentExample.py
def get_artifact_id_from_experiment(
    mlops_client,
    dai_experiment
):
    # Looking for the ID of the artifact to deploy.
    artifact_list = mlops_client.storage.artifact.list_entity_artifacts(
        mlops.StorageListEntityArtifactsRequest(entity_id=dai_experiment.key)
    ).artifact

    for ar in artifact_list:
        if ar.type == "dai/mojo_pipeline":
            return ar.id
    else:
        raise LookupError(f"Could not find 'dai/mojo_pipeline' artifact.")

Customize the composition of the deployment.

ABTestDeploymentExample.py
def create_dai_mojo_composition(experiment_id: str, artifact_id: str):
    return mlops.DeployDeploymentComposition(
        experiment_id=experiment_id,
        artifact_id=artifact_id,
        deployable_artifact_type_name="dai_mojo_pipeline",
        artifact_processor_name="dai_mojo_pipeline_extractor",
        runtime_name="dai_mojo_runtime",
    )

Set up the token provider using an existing refresh token.

Set up the Driverless AI client.

ABTestDeploymentExample.py
dai_client = driverlessai.Client(
    address=DRIVERLESS_URL, token_provider=mlops_token_provider
)

Set up the MLOps client.

Fetch the credit card dataset from S3 as the training dataset and start two experiments exp_cc_a and exp_cc_b.

ABTestDeploymentExample.py
# Fetching the credit card dataset from S3.
ds_train = dai_client.datasets.create(data=TRAIN_FILE_URL, data_source="s3")

exp_cc_a = run_dai_experiment(
    dai_client=dai_client,
    train_dataset=ds_train,
    experiment_name="cc_experiment_a",
    accuracy_setting=1,
    time_setting=1,
    interpretability_setting=1
)

exp_cc_b = run_dai_experiment(
    dai_client=dai_client,
    train_dataset=ds_train,
    experiment_name="cc_experiment_b",
    accuracy_setting=2,
    time_setting=2,
    interpretability_setting=2
)

Create a project in Driverless AI and link the fetched dataset to the project. Then link both experiments exp_cc_a and exp_cc_b to the project.

ABTestDeploymentExample.py
# Creating a project in DAI
prj = dai_client.projects.create(PROJECT_NAME)

# Linking the dataset to the project.
prj.link_dataset(ds_train, dataset_type="train_dataset")

# Linking both experiments to the project
prj.link_experiment(exp_cc_a)
prj.link_experiment(exp_cc_b)

Fetch the available deployment environments and look up the ID of the selected deployment environment.

Get the artifact IDs for the experiments.

ABTestDeploymentExample.py
artifact_id_a = get_artifact_id_from_experiment(
    mlops_client=mlops_client, dai_experiment=exp_cc_a
)

artifact_id_b = get_artifact_id_from_experiment(
    mlops_client=mlops_client, dai_experiment=exp_cc_b
)

Customize the composition of the deployment to perform an A/B Test deployment and specify the deployment as a split deployment.

ABTestDeploymentExample.py
# Customize the composition of the deployment to perform A/B Test deployment
a_element = mlops.DeploySplitElement(
    deployment_composition=create_dai_mojo_composition(
        experiment_id=exp_cc_a.key, artifact_id=artifact_id_a
    ),
    weight=50,
)

b_element = mlops.DeploySplitElement(
    deployment_composition=create_dai_mojo_composition(
        experiment_id=exp_cc_b.key, artifact_id=artifact_id_b
    ),
    weight=50,
)

# Specify the deployment as a split deployment.
to_deploy = mlops.DeployDeployment(
    project_id=prj.key,
    deployment_environment_id=deployment_env_id,
    split_deployment=mlops.DeploySplitDeployment(
        split_elements=[a_element, b_element]
    ),
)

Finally, create a deployment and wait for the deployment to become healthy. This analyzes and sets the metadata and parameters of the model, and deploys it to the DEV environment.

Feedback

Submit and view feedback for this page
Send feedback about H2O MLOps to cloud-feedback@h2o.ai

A/B Test deployment

Example walkthrough​

Example walkthrough