A/B Test deployment

This example demonstrates how you can deploy an A/B Test deployment in MLOps using Driverless AI. It creates one dataset, two experiments (experiment A and experiment B), and one project. The dataset and experiments are then linked to the project and deployed to the DEV environment in MLOps.

Before you begin

• Install Driverless AI.
• Launch a new instance of Driverless AI.
• You will need the values for the following constants in order to successfully carry out the task. Contact your administrator to obtain deployment specific values.

Constant	Value	Description
MLOPS_API_URL	Usually: https://api.mlops.my.domain	Defines the URL for the MLOps Gateway component. You can verify the correct URL by navigating to the API URL in your browser. It should provide a page with a list of available routes.
TOKEN_ENDPOINT_URL	https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token	Defines the token endpoint URL of the Identity Provider. This uses Keycloak as the Identity Provider. Keycloak Realm should be provided.
REFRESH_TOKEN	<your-refresh-token>	Defines the user's refresh token
CLIENT_ID	<your-client-id>	Sets the client id for authentication. This is the client you will be using to connect to MLOps.
PROJECT_NAME	ABTestDeploymentExample	Defines a project name that the script will be using.
DRIVERLESS_URL	<your-dai-instance-url>	Defines the URL of the Driverless AI instance you have launched.
TRAIN_FILE_URL	s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv	Defines the example training dataset.
TARGET_COLUMN	DEFAULT_PAYMENT_NEXT_MONTH	Defines the target column for the experiment.
DEPLOYMENT_ENVIRONMENT	DEV	Defines the target deployment environment.
REFRESH_STATUS_INTERVAL	1.0	Defines a refresh interval for the deployment health check.
MAX_WAIT_TIME	300	Defines maximum waiting time for the deployment to become healthy.

The following steps demonstrate how you can use MLOps Python client to deploy an A/B Test deployment.

1. Download the ABTestDeploymentExample.py file.

2. Change the values of the following constants in your ABTestDeploymentExample.py file as given in the preceding data table.

   ABTestDeploymentExample.py

   ### Constants
   MLOPS_API_URL = <MLOPS_API_URL>
   TOKEN_ENDPOINT_URL = <TOKEN_ENDPOINT_URL>
   REFRESH_TOKEN = <REFRESH_TOKEN>
   CLIENT_ID = <CLIENT_ID>
   PROJECT_NAME = <PROJECT_NAME>
   DRIVERLESS_URL = <DRIVERLESS_URL>
   TRAIN_FILE_URL = <TRAIN_FILE_URL>
   TARGET_COLUMN = <TARGET_COLUMN>
   DEPLOYMENT_ENVIRONMENT = <DEPLOYMENT_ENVIRONMENT>
   REFRESH_STATUS_INTERVAL = <REFRESH_STATUS_INTERVAL>
   MAX_WAIT_TIME = <MAX_WAIT_TIME>

   ABTestDeploymentExample.py

   ### Constants
   MLOPS_API_URL = "https://api.mlops.my.domain"
   TOKEN_ENDPOINT_URL="https://mlops.keycloak.domain/auth/realms/[fill-in-realm-name]/protocol/openid-connect/token"
   REFRESH_TOKEN="<your-refresh-token>"
   CLIENT_ID="<your-mlops-client>"
   DRIVERLESS_URL = "<your-dai-instance-url>"
   PROJECT_NAME = "ABTestDeploymentExample"
   TRAIN_FILE_URL = "s3://h2o-public-test-data/smalldata/kaggle/CreditCard/creditcard_train_cat.csv"
   TARGET_COLUMN = "DEFAULT_PAYMENT_NEXT_MONTH"
   DEPLOYMENT_ENVIRONMENT = "DEV"
   REFRESH_STATUS_INTERVAL = 1.0
   MAX_WAIT_TIME = 300

3. Run the ABTestDeploymentExample.py file.

   python3 ABTestDeploymentExample.py

   Complete 100.00% - [4/4] Computed stats for column DEFAULT_PAYMENT_NEXT_MONTH
   Experiment launched at: <your-dai-url>/#/experiment?key=a4b3e9f4-12f9-11ed-a7d2-1e7ead84e80b
   Complete 100.00% - Status: Complete                                           
   Experiment launched at: <your-dai-url>/#/experiment?key=e8e08542-12f9-11ed-a7d2-1e7ead84e80b
   Complete 100.00% - Status: Complete                                           
   Deployment has become healthy  

4. Navigate to the launched Driverless AI instance from your browser and click the Experiments tab. The script has created two experiments, cc_experiment_a and cc_experiment_b.

   

   Click the experiment name cc_experiment_a to view details of experiment A.

   

   Click the experiment name cc_experiment_b to view details of experiment B.

   

   Note

   For more information about experiments in Driverless AI, see Driverless AI documentation.

5. Click the Projects tab in the Driverless AI instance and select the project ABTestDeploymentExample to view the created project.

   

   Note

   For more information about projects in Driverless AI, see Driverless AI documentation.

6. Finally, navigate to MLOps and click the project name ABTestDeploymentExample under Projects to view the deployments.

   

   Note

   For more information about model deployments in MLOps, see Understand deployments in MLOps.

Example walkthrough

The following steps provide a walkthrough of each of the sections in the ABTestDeploymentExample.py file.

1. Include the helper function, which waits for the deployment to be healthy.

2. Include the following function run_dai_experiment, which starts a Driverless AI experiment.

   ABTestDeploymentExample.py

   def run_dai_experiment(
       dai_client: driverlessai.Client,
       train_dataset,
       experiment_name: str,
       accuracy_setting: int,
       time_setting: int,
       interpretability_setting: int,
       task: str = "classification",
       scorer: str = "AUC"
   ):
       # Starting a basic credit card experiment with minimal parameters.
       exp_cc = dai_client.experiments.create(
           name=experiment_name,
           train_dataset=train_dataset,
           target_column=TARGET_COLUMN,
           task=task,
           accuracy=accuracy_setting,
           time=time_setting,
           interpretability=interpretability_setting,
           scorer=scorer,
           enable_gpus=False,
           seed=1234,
           cols_to_drop=["ID"],
       )
   
       return exp_cc

3. Include the following function to get the artifact ID from an experiment in order to deploy it.

   ABTestDeploymentExample.py

   def get_artifact_id_from_experiment(
       mlops_client,
       dai_experiment
   ):
       # Looking for the ID of the artifact to deploy.
       artifact_list = mlops_client.storage.artifact.list_entity_artifacts(
           mlops.StorageListEntityArtifactsRequest(entity_id=dai_experiment.key)
       ).artifact
   
       for ar in artifact_list:
           if ar.type == "dai/mojo_pipeline":
               return ar.id
       else:
           raise LookupError(f"Could not find 'dai/mojo_pipeline' artifact.")

4. Customize the composition of the deployment.

   ABTestDeploymentExample.py

   def create_dai_mojo_composition(experiment_id: str, artifact_id: str):
       return mlops.DeployDeploymentComposition(
           experiment_id=experiment_id,
           artifact_id=artifact_id,
           deployable_artifact_type_name="dai_mojo_pipeline",
           artifact_processor_name="dai_mojo_pipeline_extractor",
           runtime_name="dai_mojo_runtime",
       )

5. Set up the token provider using an existing refresh token.

6. Set up the Driverless AI client.

   ABTestDeploymentExample.py

   dai_client = driverlessai.Client(
       address=DRIVERLESS_URL, token_provider=mlops_token_provider
   )

7. Set up the MLOps client.

8. Fetch the credit card dataset from S3 as the training dataset and start two experiments exp_cc_a and exp_cc_b.

   ABTestDeploymentExample.py

   # Fetching the credit card dataset from S3.
   ds_train = dai_client.datasets.create(data=TRAIN_FILE_URL, data_source="s3")
   
   exp_cc_a = run_dai_experiment(
       dai_client=dai_client,
       train_dataset=ds_train,
       experiment_name="cc_experiment_a",
       accuracy_setting=1,
       time_setting=1,
       interpretability_setting=1
   )
   
   exp_cc_b = run_dai_experiment(
       dai_client=dai_client,
       train_dataset=ds_train,
       experiment_name="cc_experiment_b",
       accuracy_setting=2,
       time_setting=2,
       interpretability_setting=2
   )

9. Create a project in Driverless AI and link the fetched dataset to the project. Then link both experiments exp_cc_a and exp_cc_b to the project.

   ABTestDeploymentExample.py

   # Creating a project in DAI
   prj = dai_client.projects.create(PROJECT_NAME)
   
   # Linking the dataset to the project.
   prj.link_dataset(ds_train, dataset_type="train_dataset")
   
   # Linking both experiments to the project
   prj.link_experiment(exp_cc_a)
   prj.link_experiment(exp_cc_b)

10. Fetch the available deployment environments and look up the ID of the selected deployment environment.

11. Get the artifact IDs for the experiments.

    ABTestDeploymentExample.py

    artifact_id_a = get_artifact_id_from_experiment(
        mlops_client=mlops_client, dai_experiment=exp_cc_a
    )
    
    artifact_id_b = get_artifact_id_from_experiment(
        mlops_client=mlops_client, dai_experiment=exp_cc_b
    )

12. Customize the composition of the deployment to perform an A/B Test deployment and specify the deployment as a split deployment.

    ABTestDeploymentExample.py

    # Customize the composition of the deployment to perform A/B Test deployment
    a_element = mlops.DeploySplitElement(
        deployment_composition=create_dai_mojo_composition(
            experiment_id=exp_cc_a.key, artifact_id=artifact_id_a
        ),
        weight=50,
    )
    
    b_element = mlops.DeploySplitElement(
        deployment_composition=create_dai_mojo_composition(
            experiment_id=exp_cc_b.key, artifact_id=artifact_id_b
        ),
        weight=50,
    )
    
    # Specify the deployment as a split deployment.
    to_deploy = mlops.DeployDeployment(
        project_id=prj.key,
        deployment_environment_id=deployment_env_id,
        split_deployment=mlops.DeploySplitDeployment(
            split_elements=[a_element, b_element]
        ),
    )

13. Finally, create a deployment and wait for the deployment to become healthy. This analyzes and sets the metadata and parameters of the model, and deploys it to the DEV environment.

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O MLOps to cloud-feedback@h2o.ai