Skip to main content
Version: v1.7.7-1 🚧

Remote embedding models (NVIDIA NIM)

Overview​

Administrators can add embedding models served by an OpenAI-compatible /v1/embeddings endpoint, such as an NVIDIA embedding NIM. Each configured model appears in the embedding model list when users create a collection. The remote endpoint embeds documents at ingestion and questions at retrieval time, so these models need no GPU or local model download.

The embedding model list shows no remote models until an administrator configures them. The built-in embedding models stay available.

important

With a hosted endpoint, Enterprise h2oGPTe sends document chunks, user questions, and generated answers to that endpoint. It sends questions without PII redaction, either as typed or as rewritten for retrieval from the chat history. After the LLM answers, Enterprise h2oGPTe sends the answer and the retrieved chunks to rerank the references. Document chunks carry PII redaction only if the collection's guardrails redact PII at ingestion.

Prerequisites​

Before you configure a remote model, you need:

  • Access to the deployment configuration: the Helm values on Kubernetes, or the environment on Docker Compose.
  • An OpenAI-compatible embedding endpoint, such as an NVIDIA embedding NIM, and its API key if the endpoint requires one.

Configure remote embedding models​

Configure remote models with the H2OGPTE_CORE_REMOTE_ENCODER_MODELS environment variable. It holds a JSON object keyed by model ID. Enterprise h2oGPTe sends the key as model to the endpoint and stores it as the collection's embedding model name.

{
  "nvidia/llama-nemotron-embed-vl-1b-v2": {
    "base_url": "https://integrate.api.nvidia.com/v1",
    "api_key": "nvapi-...",
    "dimensions": 1024,
    "max_tokens": 8192,
    "batch_size": 32,
    "display_name": "NVIDIA Nemotron embed (NIM)",
    "description": "Hosted NVIDIA embedding model",
    "languages": []
  }
}
FieldRequiredMeaning
base_urlyesEndpoint base URL, ending in /v1. Enterprise h2oGPTe appends /embeddings.
api_keynoSent as Authorization: Bearer <key>. If left out, Enterprise h2oGPTe uses H2OGPTE_CORE_REMOTE_ENCODER_API_KEY when set. Leave both unset for a self-hosted NIM without authentication.
dimensionsyesVector size, 1 to 2000. Enterprise h2oGPTe sends it to the endpoint on every request.
max_tokensnoModel context in tokens (default 512). Enterprise h2oGPTe splits longer chunks and averages their embeddings.
batch_sizenoTexts per request (default 32).
timeoutnoRequest timeout in seconds (default 60).
display_name, description, languagesnoShown in the embedding model list. An empty languages list means multilingual.

Each request sends input_type: "passage" for documents, input_type: "query" for questions, truncate: "END", and encoding_format: "float". Enterprise h2oGPTe retries rate limit (429) errors, server (5xx) errors, connection errors, and timeouts twice, with a longer delay each time. Any other error fails the ingestion or the query, and so does a third failed attempt.

The 2000-dimension limit​

pgvector can't build an HNSW index on vectors wider than 2000 dimensions, so dimensions must be 2000 or less. A larger value stops the services from starting with an error. If a model supports Matryoshka embeddings (for example nvidia/llama-nemotron-embed-vl-1b-v2), pick a smaller size such as 1024. If a model only returns a single size above 2000 (for example nvidia/nemotron-3-embed-1b, which only returns 2048), you can't use it. For a model without Matryoshka support, set dimensions to its native size.

Do not change dimensions or the model behind an ID once collections use it. Existing vectors would no longer match new queries. Add a new model ID instead.

Hosted NIM (build.nvidia.com)​

  1. Create an API key on build.nvidia.com.
  2. Use https://integrate.api.nvidia.com/v1 as base_url and the key as api_key.
  3. Check which embedding models your key can reach: GET https://integrate.api.nvidia.com/v1/models.

Self-hosted NIM​

Run the embedding NIM container in your cluster, then point base_url at its service, for example http://nemotron-embed.nim.svc.cluster.local:8000/v1. Leave out api_key if the NIM does not require one.

Configure with Helm​

The setting carries API keys, so the chart reads it from an existing Secret:

kubectl create secret generic nim-embeddings \
  --from-file=remoteEncoderModels=remote-encoder-models.json
global:
  embedding:
    remoteEncoderModels:
      existingSecret: nim-embeddings
      existingSecretKey: remoteEncoderModels

The chart passes it to the core, crawl, chat, and models services.

The chart does not wire the shared H2OGPTE_CORE_REMOTE_ENCODER_API_KEY. To keep the key out of the JSON, store it in the Secret (here as apiKey in nim-embeddings) and add it to extraEnv of each of core, crawl, chat and models:

core:
  extraEnv:
    - name: H2OGPTE_CORE_REMOTE_ENCODER_API_KEY
      valueFrom:
        secretKeyRef:
          name: nim-embeddings
          key: apiKey

Repeat the same entry under crawl.extraEnv, chat.extraEnv and models.extraEnv.

Configure with Docker Compose​

Export H2OGPTE_CORE_REMOTE_ENCODER_MODELS with the JSON before starting the stack. docker-compose.h2ogpte.yml passes it to the core, crawl, chat, and models services. It doesn't pass H2OGPTE_CORE_REMOTE_ENCODER_API_KEY, so set api_key in the JSON.

Verify​

Restart the services so they read the new setting, then start creating a collection. The Embedding Model list shows each remote model under its display_name, or under its model ID when display_name is empty.

Feedback