Skip to main content
Version: v1.7.7-1 🚧

Use NVIDIA NIM for LLMs

Overview​

NVIDIA NIM for LLMs serves models such as Nemotron behind an OpenAI-compatible /v1/chat/completions API. Enterprise h2oGPTe registers a NIM like any other OpenAI-compatible model, through the LiteLLM proxy. After that, chat, RAG, guardrails, agents, and the LLM self-test can use the model.

Two deployment options use the same entry shape:

Deploymentapi_baseAPI key
Self-hosted NIM container (for example an air-gapped Dell AI Factory with NVIDIA AI Enterprise)http://<nim-host>:8000/v1Any non-empty value unless you put an auth proxy in front of the NIM
NVIDIA-hosted endpoint on build.nvidia.comhttps://integrate.api.nvidia.com/v1An nvapi-... key from build.nvidia.com
note

Register the NIM with inference_server: litellm and an openai/<model id> LiteLLM model. Enterprise h2oGPTe then sends structured-output requests, such as those from guardrails and agents, as response_format with type json_schema, which every current NIM enforces. A vllm_chat:<url> entry sends a top-level guided_json field instead. NIM 2.x ignores that field, so JSON output isn't enforced.

Prerequisites​

  • Administrator access to Enterprise h2oGPTe
  • A NIM endpoint that Enterprise h2oGPTe can reach, or an API key from build.nvidia.com
  • For the deployment route, access to the h2oGPT Helm values or llm_models_config.yaml

Entry shape​

The following example registers nvidia/nemotron-3-super-120b-a12b as nemotron-3-super-nim. The model is in the NIM support matrix, and build.nvidia.com serves it. A commented-out copy of this entry is in llm_models_config.yaml.

- base_model: nvidia/nemotron-3-super-120b-a12b
  display_name: nemotron-3-super-nim
  inference_server: litellm
  lite_llm_yaml: '{"model_name": "nemotron-3-super-nim", "litellm_params": {"model": "openai/nvidia/nemotron-3-super-120b-a12b", "api_base": "https://integrate.api.nvidia.com/v1", "api_key": "os.environ/NVIDIA_API_KEY"}}'
  max_seq_len: 262144
  tokenizer_base_model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
  images_num_max: -2
  max_output_seq_len: 32768
  is_actually_reasoning_model: true
  • model in litellm_params: openai/ followed by the model id the NIM serves. For a self-hosted NIM, use the id returned by curl http://<nim-host>:8000/v1/models (it is NIM_SERVED_MODEL_NAME when that is set).
  • model_name and display_name: keep them identical. model_name is the routing key at the LiteLLM proxy. The proxy also matches names in lowercase, so two names that differ only by case share one routing group. For example, don't name the model nvidia/nemotron-3-super-120b-a12b, which collides with the shipped OpenRouter entry nvidia/Nemotron-3-Super-120B-A12B.
  • api_key: os.environ/NVIDIA_API_KEY is read from the environment of the h2oGPT LLM service, which runs the LiteLLM proxy (see Secrets). A self-hosted NIM without auth still needs a non-empty value, for example "api_key": "not-used".
  • max_seq_len: set this to the context length your NIM actually serves (its --max-model-len), which can be lower than the model maximum.
  • tokenizer_base_model: used for token counting. In an air-gapped cluster, make this tokenizer available in the local Hugging Face cache, or point it at a tokenizer that is already there.
  • is_actually_reasoning_model: Nemotron 3 Super reasons by default. NIM returns the reasoning in a separate reasoning_content field, so content stays clean for JSON consumers.

Register the model​

From the model catalog​

The model catalog lists NVIDIA-hosted models such as Nemotron 3 Super. Go to Models, click Add model, pick the model card, enter your nvapi-... key in API key, and click Add model. For a self-hosted NIM, pick the NIM model card, enter the NIM address in Endpoint URL, and click Load models to pick the Model ID. For a model the catalog doesn't list, use a custom configuration.

From a custom configuration (no restart)​

Add the model as a runtime model: go to Models, click Add model, click Custom configuration, click Raw JSON, paste the entry, and click Add model. Click Save without testing if the NIM isn't reachable yet. See Add a custom model for the steps. A runtime model requires display_name and both cost fields:

{
  "base_model": "nvidia/nemotron-3-super-120b-a12b",
  "display_name": "nemotron-3-super-nim",
  "inference_server": "litellm",
  "lite_llm_yaml": {"model_name": "nemotron-3-super-nim", "litellm_params": {"model": "openai/nvidia/nemotron-3-super-120b-a12b", "api_base": "http://nim-llm.nim.svc:8000/v1", "api_key": "os.environ/NVIDIA_API_KEY"}},
  "max_seq_len": 262144,
  "max_output_seq_len": 32768,
  "images_num_max": -2,
  "is_actually_reasoning_model": true,
  "cost_per_1k_input_tokens": 0,
  "cost_per_1k_output_tokens": 0
}

A cost of 0 is fine for a self-hosted NIM, but it bills nothing, so the per-user cost limits never apply to the model. For build.nvidia.com, set real prices.

To script it, send the same object to POST /api/v1/llm-registry/admin/runtime-models, or pass it to the Python client method create_runtime_llm. See Python client examples.

From the deployment configuration​

  • Docker Compose / local: add the entry to llm_models_config.yaml and run make generate-llm-config.
  • Helm: add the entry to the h2oGPT chart's model lock:
llm:
  externalLLM:
    enabled: true
    modelLock:
      - base_model: nvidia/nemotron-3-super-120b-a12b
        display_name: nemotron-3-super-nim
        inference_server: litellm
        lite_llm_yaml: '{"model_name": "nemotron-3-super-nim", "litellm_params": {"model": "openai/nvidia/nemotron-3-super-120b-a12b", "api_base": "http://nim-llm.nim.svc:8000/v1", "api_key": "os.environ/NVIDIA_API_KEY"}}'
        max_seq_len: 262144
        max_output_seq_len: 32768
        images_num_max: -2
        is_actually_reasoning_model: true

Secrets​

Put the key in the h2oGPT LLM pod's environment, for example with the h2oGPT chart:

llm:
  secret:
    NVIDIA_API_KEY: "nvapi-..."

Verify​

  1. Go to Models and confirm the model is listed.
  2. Click Run self-tests, select Quick test, and pick the model. For a reasoning model such as this one, the quick test checks plain chat only and skips its JSON leg. To check structured output, send a chat request with a JSON schema.
  3. Ask a question in a chat, and in a chat with a collection.

Known caveats​

The following behavior comes from tests of nvidia/nemotron-3-super-120b-a12b on integrate.api.nvidia.com and from the NIM documentation. Self-hosted NIM behavior depends on the NIM release.

TopicBehavior
Structured outputresponse_format with type json_schema works on hosted and on NIM 2.x. A top-level guided_json works on hosted and on NIM 1.x, but NIM 2.x ignores it. nvext.guided_json (NIM 1.0 to 1.8) returns HTTP 400 on hosted, and NIM 2.x removed the nvext object.
Unknown request fieldsThe hosted endpoint returns HTTP 400 Unsupported parameter(s). A self-hosted NIM 2.x ignores them.
Sampling parametersThe hosted endpoint accepts temperature (including 0), top_p, top_k, repetition_penalty, presence_penalty, frequency_penalty and seed. logit_bias is accepted. Nemotron 3 Super rejects min_p with HTTP 400 because it uses speculative decoding. To drop a parameter an endpoint rejects, set unsupported_sampling_params on the entry, for example unsupported_sampling_params: top_k.
max_tokensReasoning tokens count toward max_tokens. The hosted API reference lists 32768 as the maximum for Nemotron 3 Super, so keep max_output_seq_len at or below that.
stream_optionsRejected unless stream is true.
Reasoning controlThe hosted endpoint accepts reasoning_effort (none turns reasoning off) and chat_template_kwargs.enable_thinking. On a self-hosted NIM 2.x, vLLM maps reasoning_effort to enable_thinking. Leave supports_reasoning_effort unset unless you want h2oGPTe to send it.
Feedback