Skip to main content
Version: v1.2.0

Deploy and chat with LLM deployments

Large language model (LLM) deployments run streaming model types served by the H2O MLOps vLLM runtimes. These deployments expose the standard OpenAI API at /v1. This page shows how to deploy an LLM and chat with it using the Python client, the OpenAI SDK, or curl.

Alpha feature

Chat/OpenAI-compatible scoring is an alpha feature and may change or be removed in future releases. The Python client emits a UserWarning on first use.

Prerequisites​

Before you begin,

  1. Import the necessary Python packages. For instructions, see Step 1: Import the required packages.
  2. Connect to H2O MLOps. For instructions, see Connect to H2O MLOps.
  3. Create a workspace. For instructions, see Create a workspace.
  4. Create one or two experiments. For instructions, see Create an experiment.
  5. Create models and register the experiments with them. For instructions, see Register an experiment with a model.
  6. Create a deployment. For instructions, see Create a deployment.

This tutorial also uses the following packages:

pip install h2o-mlops huggingface_hub openai

The examples on this page assume a connected client and a workspace, as covered in the preceding prerequisites:

import h2o_mlops

client = h2o_mlops.Client()
workspace = client.workspaces.create(name="llm-demo")

Choose a runtime​

H2O MLOps provides two runtimes for LLMs:

RuntimeModel formatHardware
vllm_runtimeSafetensorsCPU by default; GPU when you request one
vllm_gguf_runtimeGGUFGPU only

Deploy a safetensors model on vllm_runtime if you don't have a GPU.

Tune the runtime​

The vLLM runtimes read the following environment variables. For more information, see Environment variables.

VariablePurpose
VLLM_CPU_KVCACHE_SPACESize of the key-value cache, in GiB, on CPU deployments. Defaults to 4. Has no effect on vllm_gguf_runtime, which runs only on GPU.
VLLM_TOKENIZERHugging Face repository to load the tokenizer from, instead of the model artifact. The deployment resolves this at startup, so it needs network access to Hugging Face.
VLLM_EXTRA_ARGSSpace-separated vllm serve flags that H2O MLOps doesn't surface, for example --max-model-len 8192. Values that contain spaces aren't supported.

Pass them to deployments.create() as environment_variables:

workspace.deployments.create(
   # ... name, composition_options, and security_options as in the sections below.
   environment_variables={"VLLM_CPU_KVCACHE_SPACE": "8"},
)

Add this argument to the deployments.create() calls in the walkthroughs that follow. To change a variable on a deployment that already exists, pass environment_variables to its update() method.

Download a GGUF model from Hugging Face​

Download a single-file GGUF. Multi-part GGUFs aren't supported: if the model artifact contains more than one .gguf file, the deployment fails at startup with ERROR: multiple .gguf files in MODEL_PATH — single-file GGUF only:. This example uses a small, ungated model:

from huggingface_hub import hf_hub_download

hf_hub_download(
   repo_id="Qwen/Qwen2.5-0.5B-Instruct-GGUF",
   filename="qwen2.5-0.5b-instruct-q4_k_m.gguf",
   local_dir="./llm_model",
)

vLLM can't serve a bare .gguf. It reads the model configuration from the config.json next to the GGUF file, and without it the deployment fails at startup with Unrecognized model ... should have a model_type key in its config.json. Download the configuration and the tokenizer from the repository the GGUF was quantized from into the same directory, and exclude that repository's weights:

from huggingface_hub import snapshot_download

snapshot_download(
   repo_id="Qwen/Qwen2.5-0.5B-Instruct",
   local_dir="./llm_model",
   allow_patterns=["*.json", "tokenizer*", "*.model", "merges.txt", "vocab.json"],
   ignore_patterns=["*.safetensors", "*.bin", "*.pt", "*.h5", "*.gguf"],
)

./llm_model now holds the .gguf file alongside config.json and the tokenizer files, which makes the artifact self-contained. Use the GGUF's source model repository: a tokenizer from a different model produces incorrect output. Because the directory contains no *.safetensors files, H2O MLOps still sets the artifact type to GGUF.

Alternatively, set VLLM_TOKENIZER to the source repository and the deployment loads the tokenizer from Hugging Face at startup instead of from the artifact. This needs network access to Hugging Face from the cluster, and it supplies only the tokenizer, so the artifact still needs config.json. For more information, see Tune the runtime.

note

The Hugging Face download happens only on your machine. The Python client then uploads the model to H2O MLOps as an artifact, and the runtime never pulls model weights from Hugging Face directly.

For gated models, request access on the model page, then create a token at huggingface.co/settings/tokens and pass token=os.environ["HF_TOKEN"] to the download function.

Zip the GGUF model for upload​

Zip the ./llm_model directory. H2O MLOps detects the .gguf file and sets the artifact type automatically.

import os
import zipfile


def zip_dir(src_dir: str, zip_path: str, compression=zipfile.ZIP_DEFLATED) -> str:
   with zipfile.ZipFile(zip_path, "w", compression) as zf:
       for root, _, files in os.walk(src_dir):
           if os.sep + ".cache" in root:
               continue  # skip huggingface_hub metadata
           for f in files:
               if f.startswith("."):
                   continue
               full = os.path.join(root, f)
               zf.write(full, os.path.relpath(full, src_dir))
   return zip_path


zip_dir("./llm_model", "llm_model.zip")

Deploy a GGUF model on vllm_gguf_runtime​

Create the experiment and model, then register the experiment with the model:

experiment = workspace.experiments.create(data="llm_model.zip", name="qwen-gguf")
model = workspace.models.create(name="qwen-0.5b-gguf")
model.register(experiment=experiment)

Find the vLLM GGUF scoring runtime:

runtime = next(
   (r for r in client.runtimes.scoring.list()
    if r.runtime.uid.endswith("/vllm_gguf_runtime")),
   None,
)
if runtime is None:
   raise LookupError("vllm_gguf_runtime isn't available on this cluster.")

vllm_gguf_runtime runs only on GPU nodes, so you must request a GPU and target your cluster's GPU nodes. If the deployment schedules onto a CPU node, the container fails at startup when vLLM initializes CUDA and finds no GPU. H2O MLOps publishes the runtime image for linux/amd64 only, so it can't run on other architectures. To serve a model without a GPU, use a safetensors model on vllm_runtime instead.

The affinity and toleration are shortcut names your cluster defines, often gpu. They're cluster-specific, and the client rejects a name your cluster doesn't define with a ValueError, so list the allowed ones first:

client.configs.allowed_k8s_affinities
client.configs.allowed_k8s_tolerations

Each table lists a display label in the name column and the value you pass in the uid column. For more information, see Node affinity and toleration.

Create the deployment:

from h2o_mlops.options import CompositionOptions, KubernetesOptions, SecurityOptions
from h2o_mlops.types import SecurityType

deployment = workspace.deployments.create(
   name="qwen-llm",
   composition_options=CompositionOptions(
       model=model, scoring_runtime=runtime, model_version="latest",
   ),
   security_options=SecurityOptions(
       security_type=SecurityType.HASHED_PASSPHRASE,
       passphrase="my-passphrase",
   ),
   kubernetes_options=KubernetesOptions(
       limits={"memory": "16Gi", "nvidia.com/gpu": "1"},
       affinity="gpu",
       toleration="gpu",
   ),
)

deployment.wait_for_healthy(timeout=600, interval=10)

An LLM deployment takes longer to start than a tabular one, so raise timeout from its 60-second default. wait_for_healthy() raises MLOpsDeploymentError if the deployment enters FAILED, and TimeoutError if it doesn't become healthy in time.

If wait_for_healthy() raises either exception, check deployment.state and read deployment.logs() for the startup error. The logs are keyed by pod, and the startup error is under the key ending in .runtime. A GGUF model on a CPU node crash-loops at startup. For more information, see View deployment logs.

caution

LLM (streaming) model types support single-model deployments only. deployments.create() rejects champion/challenger and A/B test modes for these model types with a ValueError. Monitoring doesn't apply to LLM deployments.

Retrieve a deployment scorer​

Retrieve the deployment's scorer and check its chat support. For all scorer properties, see Deployment scorer properties.

scorer = deployment.scorer
scorer.supports_chat        # True for LLM deployments
scorer.chat_endpoint        # https://.../<deployment-id>/v1/chat/completions
scorer.openai_base_url      # https://.../<deployment-id>/v1

Find the served model name​

Every request must name the model it targets. The runtime serves the model under a name that H2O MLOps assigns, and rejects any other value, so retrieve that name once and reuse it. The Python client has no helper for this, so query the deployment's /v1/models endpoint with the OpenAI SDK:

from openai import OpenAI

oai = OpenAI(base_url=scorer.openai_base_url, api_key="my-passphrase")
model_id = oai.models.list().data[0].id

For an OIDC-secured deployment, pass token_provider.token() from your h2o_authn.TokenProvider as api_key. The client keeps the token you pass at construction and doesn't refresh it the way scorer.chat() does, so build a new client if the token expires during a long session.

Chat with the deployment using the Python client​

Send a non-streaming request, which returns the full OpenAI response dict:

resp = scorer.chat(
   [{"role": "user", "content": "Say hello in one short sentence."}],
   auth_value="my-passphrase",
   model=model_id,
   max_tokens=64,
)
print(resp["choices"][0]["message"]["content"])

Send a streaming request, which returns an iterator of text deltas:

for chunk in scorer.chat(
   [{"role": "user", "content": "Count from one to five."}],
   auth_value="my-passphrase",
   model=model_id,
   stream=True,
):
   print(chunk, end="", flush=True)

The chat() method accepts the following parameters: messages, auth_value, stream, model, max_tokens, temperature, and timeout.

  • Always pass model. The default, "local", isn't a name the vLLM runtime serves, and the request fails with a 404. For more information, see Find the served model name.
  • The client sends auth_value as an Authorization: Bearer token. For an OIDC-secured deployment, you can omit auth_value; the client uses its own token automatically.
  • The default timeout is None, which means no read timeout. LLM generation can be slow, especially on CPU.
  • With stream=True, chat() returns an iterator of assistant text deltas (strings), not raw OpenAI chunk objects. To work with raw chunk objects, use the OpenAI SDK against openai_base_url instead. See Use any OpenAI-compatible tool.
  • Connection, authentication, and HTTP errors surface on the first iteration of the returned iterator, not at the call site.
  • On a non-LLM (tabular) deployment, supports_chat, chat_endpoint, openai_base_url, and chat() all raise MLOpsEndpointError. Use score() for those deployments instead. For more information, see Score against the deployment.

Use any OpenAI-compatible tool​

Because the deployment serves the standard OpenAI API, the OpenAI SDK works out of the box. This example reuses the model_id from Find the served model name:

from openai import OpenAI

oai = OpenAI(base_url=scorer.openai_base_url, api_key="my-passphrase")
resp = oai.chat.completions.create(
   model=model_id,
   messages=[{"role": "user", "content": "Say hello."}],
)

You can also chat with the deployment using curl. This example uses jq to read the served model name:

MODEL_ID=$(curl -s -H "Authorization: Bearer my-passphrase" \
 https://<your-mlops-domain>/<deployment-id>/v1/models | jq -r '.data[0].id')

curl -N \
 -H "Authorization: Bearer my-passphrase" \
 -H "Content-Type: application/json" \
 -d "{\"model\": \"$MODEL_ID\", \"messages\": [{\"role\": \"user\", \"content\": \"hi\"}], \"stream\": true}" \
 https://<your-mlops-domain>/<deployment-id>/v1/chat/completions

Deploy a safetensors model on vllm_runtime​

You can deploy a Hugging Face safetensors model on vllm_runtime using the same upload-and-deploy flow, with three differences: download the full model snapshot, zip it without compression, and deploy on vllm_runtime. Unlike vllm_gguf_runtime, vllm_runtime runs on CPU by default and uses a GPU when you request one, so a GPU is optional.

from huggingface_hub import snapshot_download

snapshot_download(
   repo_id="h2oai/h2o-danube3-500m-chat",
   local_dir="./danube_model",
   token=os.environ.get("HF_TOKEN"),  # only needed for gated models
   allow_patterns=["*.json", "*.safetensors", "tokenizer*", "*.model"],
)

# safetensors don't compress -- use ZIP_STORED to save time.
zip_dir("./danube_model", "danube_model.zip", compression=zipfile.ZIP_STORED)

experiment = workspace.experiments.create(data="danube_model.zip", name="danube-chat")
model = workspace.models.create(name="h2o-danube3-chat")
model.register(experiment=experiment)

runtime = next(
   (r for r in client.runtimes.scoring.list()
    if r.runtime.uid.endswith("/vllm_runtime")),
   None,
)
# ... deploy and chat exactly as in the previous sections.

To run this model on a GPU, pass the kubernetes_options shown in Deploy a GGUF model on vllm_gguf_runtime. To run it on CPU, omit kubernetes_options or set memory limits only.

This deployment has its own scorer and its own served model name. Repeat Retrieve a deployment scorer and Find the served model name against it before you chat.

CPU performance

On CPU, vLLM reserves 4 GiB for the key-value cache on top of the model weights. To change it, set VLLM_CPU_KVCACHE_SPACE. For more information, see Tune the runtime. Generation on CPU runs at a small fraction of GPU throughput.

Size the deployment for the weights plus the cache: a 4-billion-parameter model needs roughly 16 GiB of RAM. Expect a multi-minute startup while vLLM initializes its engine and loads the weights, and prefer a GPU for production workloads.

Clean up​

Delete the deployment when you're done:

deployment.delete()

deployment refers to the last one you created. If you followed both walkthroughs, delete each one separately: the GGUF deployment holds a GPU until you do.

Feedback