On-demand mode

On-demand mode rests a deployment at zero replicas while no requests arrive, then wakes it to its full configured replica count when the next request comes in.

Use it for deployments with unpredictable or infrequent traffic, where idle capacity costs more than the delay of a cold start. The cost lands on the first request after an idle period: it waits while the deployment starts.

For how on-demand mode relates to scheduled scaling and scheduled target switching, see Scheduling overview.

Prerequisites

Before you configure on-demand mode, know two things:

• An administrator must turn it on. On-demand mode runs on KEDA and the KEDA HTTP add-on. An administrator installs both and enables on-demand mode on the H2O MLOps server. For more information, see Configure scheduling and on-demand mode.
• On-demand mode has no UI. Configure it through the H2O MLOps API or the Python client. The H2O MLOps UI has no on-demand controls.

How it works

With on-demand mode enabled, the deployment's route runs through an interceptor proxy. Once the deployment goes idle_timeout without a request, H2O MLOps scales it to zero replicas. The next request reaches the proxy, which holds it while the workload starts and then forwards it to the deployment.

Callers send the same request to the same scoring URL whether the deployment runs or sits at zero replicas. The only difference they see is the wait on that first request.

Settings

On-demand mode has three settings:

Setting	Meaning	Default	Bound
enabled	Scales the deployment to zero when it goes idle and wakes it on the next request.	false	Requires server support.
idle_timeout	Time without requests before the deployment scales to zero.	30m	At least 5m. No upper limit.
cold_start_timeout	How long the proxy holds the first request after idle while the deployment wakes.	30s	At most 2m.

Three rules are easy to get wrong:

• Timeouts must use whole seconds. Storage is second-precision, so H2O MLOps rejects a fractional value instead of rounding it. It rejects negative values too.
• Unset means "use the server default." It does not mean "no timeout." The defaults are 30m and 30s unless your administrator changed them.
• Neither bound is adjustable. idle_timeout cannot go below 5m, and cold_start_timeout cannot exceed 2m. Both bounds set fixed server policy: no deployment can override them, neither takes an environment variable, and the H2O MLOps Helm chart exposes no values for them. See Configure scheduling and on-demand mode.

Cold starts

The first request after a deployment scales to zero is slow. It waits while the deployment starts, so size cold_start_timeout to cover start-up time. Image pull, runtime start, and model load all count against it. A model that takes 45 seconds to load needs more than the 30s default.

Past cold_start_timeout, the request fails with a retryable error and the wake continues. The deployment still comes up. Treat that error as a signal to retry, not as a failed deployment.

caution

Because cold_start_timeout cannot exceed the 2m server limit, a deployment that takes longer than that to start returns the retryable error on every first request after an idle period. If your callers cannot retry, keep replicas warm through your traffic hours with scheduled scaling instead.

Limitations

• Shadow deployments. You cannot enable on-demand mode on a shadow deployment.
• Server support. Enabling on-demand mode on a server that does not have it returns FAILED_PRECONDITION with the message on-demand mode is not enabled on this server.
• Paused deployments. A paused deployment does not scale to zero or wake on request, and H2O MLOps refuses on-demand configuration changes until you resume it. Pausing does not turn off on-demand mode: resuming restores your settings.

Configure on-demand mode with the Python client

Retrieve the deployment, then configure on-demand mode through its on_demand property:

from datetime import timedelta

deployment = workspace.deployments.get(uid="48ece40f-8608-473a-92a6-388e164e9952")

deployment.on_demand.update(
   enabled=True,
   idle_timeout=timedelta(minutes=15),
   cold_start_timeout=timedelta(seconds=45),
)

The update() method takes the following arguments, all optional:

• enabled: Whether the deployment scales to zero when it goes idle and wakes on the next request.
• idle_timeout: A timedelta for the time without requests before the deployment scales to zero. Pass None to reset it to the server default.
• cold_start_timeout: A timedelta for how long the proxy holds the first request after idle. Pass None to reset it to the server default.

update() applies a partial update. Arguments you omit keep their current values.

On-demand properties

The on_demand property reports the current configuration:

• enabled: Whether on-demand mode is active for the deployment.
• idle_timeout: The configured idle timeout, or None when the deployment uses the server default.
• cold_start_timeout: The configured cold start timeout, or None when the deployment uses the server default.
• last_modified_time: When the configuration last changed.

The client caches these values after first access. To re-pull the configuration after something else changes it:

deployment.on_demand.refresh()

Turn off or delete the configuration

update(enabled=False) and clear() both stop a deployment from scaling to zero, but they leave different state behind.

To turn off on-demand mode and keep the stored timeouts:

deployment.on_demand.update(enabled=False)

Call update(enabled=True) later and the deployment picks up the same idle_timeout and cold_start_timeout.

To delete the configuration:

deployment.on_demand.clear()

clear() removes the configuration entirely, stored timeouts included. Enabling on-demand mode again starts from the server defaults.

Combine on-demand mode with scheduled scaling

A deployment can use on-demand mode and scheduled scaling at the same time, under one rule: with on-demand mode enabled, every schedule window's desired_replicas must be greater than or equal to the deployment's configured replica count. Windows become keep-warm floors. They raise the replica count for the length of the window and never lower it, because idle scale-down belongs to on-demand mode.

The guard works in both directions: whichever side you update second is the one H2O MLOps rejects. Enabling on-demand mode on a deployment that already has a window below its configured replica count fails, and so does adding such a window to a deployment that already uses on-demand mode.

Raising the deployment's configured replica count outside window hours also succeeds, and any window left below the new count stops raising anything, because the configured count is already the higher floor.

This combination supports one pattern in particular: keep replicas warm through business hours so daytime callers never pay a cold start, and let the deployment scale to zero outside them.

For a deployment configured with three replicas:

from datetime import timedelta

from h2o_mlops import options

deployment.on_demand.update(
   enabled=True,
   idle_timeout=timedelta(minutes=10),
)

deployment.schedule.update(
   enabled=True,
   windows=[
       options.ScheduledWindowOptions(
           name="business-hours",
           start_cron="0 9 * * 1-5",
           end_cron="0 17 * * 1-5",
           desired_replicas=3,
           timezone="Europe/Berlin",
       ),
   ],
)

From 09:00 to 17:00 on weekdays, all three replicas stay up. Outside the window, on-demand mode takes over: the deployment scales to zero after 10 minutes without requests and wakes on the next one. For cron syntax, window validation rules, and presets, see Scheduled scaling for deployments.

note

H2O MLOps rejects a window with desired_replicas=0 on a deployment with on-demand mode enabled. Deployments that use scheduled scaling alone keep the full range of behavior, including scale-to-zero windows.

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O MLOps to cloud-feedback@h2o.ai