Skip to main content
Version: v0.64.0

Test Time Augmentation (TTA) support

Driverless AI supports rolling-window-based predictions for time series experiments using Test Time Augmentation (TTA). TTA is only available for Python Scoring Pipeline artifacts. This page describes support for TTA in H2O MLOps.

  1. Enable TTA when deploying a model:

    If the Driverless AI Python scoring pipeline artifact type is selected when deploying a model, Test Time Augmentation will automatically be enabled for capable models.

  2. Check if the deployment has TTA support in a curl request: (optional)

    The following is a sample curl request and response to check depoyment capabilities of a Test Time Augmentation enabled deployment:

    curl -X GET https://<DEPLOYMENT_URL>/model/capabilities

    ["SCORE","CONTRIBUTION_ORIGINAL","CONTRIBUTION_TRANSFORMED","TEST_TIME_AUGMENTATION"]

    Note: TEST_TIME_AUGMENTATION must be present in the cpabilities response for Test Time Augmentation scoring to work.

  1. Score in a curl request (optional):

    TTA requires passing known historical target values in the request.

    The following is a sample curl scoring request and response that will trigger TTA (in this example we are forecasting weekly sales):

    curl -X POST -H "Content-Type: application/json" -d @- DEPLOYMENT_ENDPOINT_URL << EOF
    {
    "fields": [
        "Store", 
        "Dept", 
        "Date", 
        "IsHoliday", 
        "Weekly_Sales"
    ],
    "rows": [
        ["1", "1", "2011-06-24", "0", "15682.81"],
        ["1", "1", "2011-07-01", "0", "15363.5"],
        ["1", "1", "2011-07-08", "0", "16148.87"],
        ["1", "1", "2011-07-15", "0", "15654.85"],
        ["1", "1", "2011-07-22", "0", "15766.6"],
        ... <intermediate rows omitted> ...
        ["1", "1", "2012-09-28", "0", "18947.81"],
        ["1", "1", "2012-10-05", "0", "21904.47"],
        ["1", "1", "2012-10-12", "0", "22764.01"],
        ["1", "1", "2012-10-19", "0", "24185.27"],
        ["1", "1", "2012-10-26", "0", "27390.81"],
        ["1", "1", "2012-11-02", "0", ""],
        ["1", "1", "2012-11-09", "0", ""]
    ], "includeFieldsInOutput": ["Store", "Dept", "Date"]
    }
    EOF

    {
    "fields": 
    [
        "Weekly_Sales",
        "Weekly_Sales.lower",
        "Weekly_Sales.upper",
        "Store",
        "Dept",
        "Date",
    ],
    "id": "70e97f86-133e-11ed-9b13-9e9aacfaa41c",
    "score": [
        ["15850.026", "-4002.4613906250015", "34435.428949218745", "1", "1", "2011-06-24"],
        ["16170.747", "-3681.7406875000015", "34756.149652343745", "1", "1", "2011-07-01"],
        ["15866.074", "-3986.4135390625015", "34451.476800781245", "1", "1", "2011-07-08"],
        ["16254.93", "-3597.5580703125015", "34840.332269531245", "1", "1", "2011-07-15"],
        ["15990.216", "-3862.2719375000015", "34575.618402343745", "1", "1", "2011-07-22"],
        ... <intermediate rows omitted> ...
        ["18941.385", "-911.1029921875015", "37526.787347656245", "1", "1", "2012-09-28"],
        ["18942.018", "-910.4701796875015", "37527.420160156245", "1", "1", "2012-10-05"],
        ["21030.857", "1178.3696640624985", "39616.260003906245", "1", "1", "2012-10-12"],
        ["21969.998", "2117.5102890624985", "40555.400628906245", "1", "1", "2012-10-19"],
        ["23745.594", "3893.1059921874985", "42330.996332031245", "1", "1", "2012-10-26"],
        ["26146.97", "6294.4829453124985", "44732.373285156245", "1", "1", "2012-11-02"],
        ["13077.371", "-6775.1166640625015", "31662.773675781245", "1", "1", "2012-11-09"]
    ]
    }

    Note: In the preceding example, the last two rows contain the predictions of interest. The rest of the rows also have predictions (rather than the input value) and can be compared against the known target values to evaluate model accuracy.

Feedback