Skip to main content
Version: v0.66.1

A/B Test deployment

This example demonstrates how you can deploy an A/B Test deployment in MLOps using Driverless AI. It creates one dataset, two experiments (experiment A and experiment B), and one project. The dataset and experiments are then linked to the project and deployed to the DEV environment in MLOps.

Before you begin
  • Install Driverless AI.
  • Launch a new instance of Driverless AI.
  • You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.
ConstantValueDescription
MLOPS_API_URLUsually: https://api.mlops.my.domainDefines the URL for the MLOps Gateway component. You can verify the correct URL by navigating to the API URL in your browser. It should provide a page with a list of available routes.
TOKEN_ENDPOINT_URL
https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token
Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided.
REFRESH_TOKEN<your-refresh-token>Defines the user's refresh token
CLIENT_ID<your-client-id>Sets the client id for authentication. This is the client you will be using to connect to MLOps.
PROJECT_NAMEABTestDeploymentExampleDefines a project name that the script will be using.
DRIVERLESS_URL<your-dai-instance-url>Defines the URL of the Driverless AI instance you have launched.
TRAIN_FILE_URL
s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv
Defines the example training dataset.
TARGET_COLUMNDEFAULT_PAYMENT_NEXT_MONTHDefines the target column for the experiment.
DEPLOYMENT_ENVIRONMENTDEVDefines the target deployment environment.
REFRESH_STATUS_INTERVAL1.0Defines a refresh interval for the deployment health check.
MAX_WAIT_TIME300Defines maximum waiting time for the deployment to become healthy.

The following steps demonstrate how you can use MLOps Python client to deploy an A/B Test deployment.

  1. Download the ABTestDeploymentExample.py file.

  2. Change the values of the following constants in your ABTestDeploymentExample.py file as given in the preceding data table.

    ABTestDeploymentExample.py
    ### Constants
    MLOPS_API_URL = <MLOPS_API_URL>
    TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
    REFRESH_TOKEN = <REFRESH_TOKEN>
    CLIENT_ID = <CLIENT_ID>
    PROJECT_NAME = <PROJECT_NAME>
    DRIVERLESS_URL = <DRIVERLESS_URL>
    TRAIN_FILE_URL = <TRAIN_FILE_URL>
    TARGET_COLUMN = <TARGET_COLUMN>
    DEPLOYMENT_ENVIRONMENT = <DEPLOYMENT_ENVIRONMENT>
    REFRESH_STATUS_INTERVAL = <REFRESH_STATUS_INTERVAL>
    MAX_WAIT_TIME = <MAX_WAIT_TIME>

  3. Run the ABTestDeploymentExample.py file.

    python3 ABTestDeploymentExample.py

  4. Navigate to the launched Driverless AI instance from your browser and click the Experiments tab. The script has created two experiments, cc_experiment_a and cc_experiment_b.

    A/B Test deployment experiments

    Click the experiment name cc_experiment_a to view details of experiment A.

    A/B Test deployment experiment A

    Click the experiment name cc_experiment_b to view details of experiment B.

    A/B Test deployment experiment B

    Note

    For more information about experiments in Driverless AI, see Driverless AI documentation.

  5. Click the Projects tab in the Driverless AI instance and select the project ABTestDeploymentExample to view the created project.

    A/B Test deployment project

    Note

    For more information about projects in Driverless AI, see Driverless AI documentation.

  6. Finally, navigate to MLOps and click the project name ABTestDeploymentExample under Projects to view the deployments.

    A/B Test deployment models

    Note

    For more information about model deployments in MLOps, see Understand deployments in MLOps.

Example walkthrough​

The following steps provide a walkthrough of each of the sections in the ABTestDeploymentExample.py file.

  1. Include the helper function, which waits for the deployment to be healthy.

  2. Include the following function run_dai_experiment, which starts a Driverless AI experiment.

    ABTestDeploymentExample.py
    def run_dai_experiment(
        dai_client: driverlessai.Client,
        train_dataset,
        experiment_name: str,
        accuracy_setting: int,
        time_setting: int,
        interpretability_setting: int,
        task: str = "classification",
        scorer: str = "AUC"
    ):
        # Starting a basic credit card experiment with minimal parameters.
        exp_cc = dai_client.experiments.create(
            name=experiment_name,
            train_dataset=train_dataset,
            target_column=TARGET_COLUMN,
            task=task,
            accuracy=accuracy_setting,
            time=time_setting,
            interpretability=interpretability_setting,
            scorer=scorer,
            enable_gpus=False,
            seed=1234,
            cols_to_drop=["ID"],
        )

        return exp_cc

  3. Include the following function to get the artifact ID from an experiment in order to deploy it.

    ABTestDeploymentExample.py
    def get_artifact_id_from_experiment(
        mlops_client,
        dai_experiment
    ):
        # Looking for the ID of the artifact to deploy.
        artifact_list = mlops_client.storage.artifact.list_entity_artifacts(
            mlops.StorageListEntityArtifactsRequest(entity_id=dai_experiment.key)
        ).artifact

        for ar in artifact_list:
            if ar.type == "dai/mojo_pipeline":
                return ar.id
        else:
            raise LookupError(f"Could not find 'dai/mojo_pipeline' artifact.")

  4. Customize the composition of the deployment.

    ABTestDeploymentExample.py
    def create_dai_mojo_composition(experiment_id: str, artifact_id: str):
        return mlops.DeployDeploymentComposition(
            experiment_id=experiment_id,
            artifact_id=artifact_id,
            deployable_artifact_type_name="dai_mojo_pipeline",
            artifact_processor_name="dai_mojo_pipeline_extractor",
            runtime_name="dai_mojo_runtime",
        )

  5. Set up the token provider using an existing refresh token.

  6. Set up the Driverless AI client.

    ABTestDeploymentExample.py
    dai_client = driverlessai.Client(
        address=DRIVERLESS_URL, token_provider=mlops_token_provider
    )

  7. Set up the MLOps client.

  8. Fetch the credit card dataset from S3 as the training dataset and start two experiments exp_cc_a and exp_cc_b.

    ABTestDeploymentExample.py
    # Fetching the credit card dataset from S3.
    ds_train = dai_client.datasets.create(data=TRAIN_FILE_URL, data_source="s3")

    exp_cc_a = run_dai_experiment(
        dai_client=dai_client,
        train_dataset=ds_train,
        experiment_name="cc_experiment_a",
        accuracy_setting=1,
        time_setting=1,
        interpretability_setting=1
    )

    exp_cc_b = run_dai_experiment(
        dai_client=dai_client,
        train_dataset=ds_train,
        experiment_name="cc_experiment_b",
        accuracy_setting=2,
        time_setting=2,
        interpretability_setting=2
    )

  9. Create a project in Driverless AI and link the fetched dataset to the project. Then link both experiments exp_cc_a and exp_cc_b to the project.

    ABTestDeploymentExample.py
    # Creating a project in DAI
    prj = dai_client.projects.create(PROJECT_NAME)

    # Linking the dataset to the project.
    prj.link_dataset(ds_train, dataset_type="train_dataset")

    # Linking both experiments to the project
    prj.link_experiment(exp_cc_a)
    prj.link_experiment(exp_cc_b)

  10. Fetch the available deployment environments and look up the ID of the selected deployment environment.

  11. Get the artifact IDs for the experiments.

    ABTestDeploymentExample.py
    artifact_id_a = get_artifact_id_from_experiment(
        mlops_client=mlops_client, dai_experiment=exp_cc_a
    )

    artifact_id_b = get_artifact_id_from_experiment(
        mlops_client=mlops_client, dai_experiment=exp_cc_b
    )

  12. Customize the composition of the deployment to perform an A/B Test deployment and specify the deployment as a split deployment.

    ABTestDeploymentExample.py
    # Customize the composition of the deployment to perform A/B Test deployment
    a_element = mlops.DeploySplitElement(
        deployment_composition=create_dai_mojo_composition(
            experiment_id=exp_cc_a.key, artifact_id=artifact_id_a
        ),
        weight=50,
    )

    b_element = mlops.DeploySplitElement(
        deployment_composition=create_dai_mojo_composition(
            experiment_id=exp_cc_b.key, artifact_id=artifact_id_b
        ),
        weight=50,
    )

    # Specify the deployment as a split deployment.
    to_deploy = mlops.DeployDeployment(
        project_id=prj.key,
        deployment_environment_id=deployment_env_id,
        split_deployment=mlops.DeploySplitDeployment(
            split_elements=[a_element, b_element]
        ),
    )

  13. Finally, create a deployment and wait for the deployment to become healthy. This analyzes and sets the metadata and parameters of the model, and deploys it to the DEV environment.

Feedback