Skip to main content
Version: v1.7.7-12 🚧

CPU-only deployment

Enterprise h2oGPTe can run without GPUs. In a CPU-only deployment, the models service runs OCR, embeddings, guardrails and document layout models on CPU, and every LLM is an external endpoint that you configure. Ingestion and retrieval are slower than on a GPU deployment.

CPU-only deployment is supported from version 1.7.7.

Enable the CPU-only profile​

The Helm chart ships a values file, values-cpu-only.yaml, next to values.yaml. Add it after your own values file:

helm upgrade --install h2ogpte ./helm/h2ogpte-chart -f my-values.yaml -f values-cpu-only.yaml

The profile:

  • removes the nvidia.com/gpu request and limit from the models pod
  • runs the models service and its embedding model on CPU (coreEncoderDevice: cpu, H2OGPTE_CORE_MODELS_MUST_USE_GPU=false, CUDA_VISIBLE_DEVICES=-1)
  • does not start the GPU-only model queues (see below) through H2OGPTE_CORE_MODELS_QUEUE_FILTER
  • makes Tesseract the default OCR engine (H2OGPTE_PARSE_OCR_MODEL=tesseract on crawl, core and chat)
important

Helm replaces lists instead of merging them. If your own values already set extraEnv for models, crawl, core or chat, copy the profile's entries into your lists instead of layering the file, or your entries are dropped.

Keep h2ogpt.enabled: false and configure external LLM endpoints. Hosting LLMs inside the cluster needs GPUs.

Do not combine the profile with the modelsCpu or modelsHeavy tier deployments: the profile sets the models queue filter directly.

What is not available on CPU​

FeatureBehavior in a CPU-only deployment
PaddleOCR (paddleocr-* OCR models: Chinese, Japanese, Korean, Arabic, Cyrillic, Devanagari, Thai and other non-Latin scripts)Not available. OCR with an explicit paddleocr-* model fails quickly with an error that names the cause.
Automatic OCR model selection (auto)Language detection runs on PaddleOCR, so it is skipped and docTR is used for every page.
Handwriting OCR (mississippi-800m) and handwriting detectionNot available.
Audio and video ingestionNot available.
VLM-based language identificationNot available.
LLMs hosted in the clusterNot available. Use external LLM endpoints.

A request for a model the models service does not run fails within seconds with an error such as Model get_paddleocr_preds is not served by this models pod. The models pod /health endpoint reports "status": "degraded" and lists dead_queues when the workers for a queue could not start.

Performance​

OCR, embeddings, layout detection and guardrails all run on CPU, so expect ingestion to be several times slower than on a GPU deployment. Throughput depends on the CPU and memory you give the models pod; size models.resources for your document volume and test with representative documents.

To take embedding work off the models pod, configure a remote embedding model.


Feedback