Skip to main content
Version: v1.7.7-1 🚧

GPU monitoring

There are two complementary sources of GPU metrics.

h2oGPTe models pod metrics​

Each models pod serves Prometheus metrics at /metrics-prometheus on its service port (8000 by default). Besides jobs_queue_size, it reports one series per GPU visible to the pod, labelled with gpu (index), uuid and name:

MetricUnit
h2ogpte_gpu_utilization_percent%
h2ogpte_gpu_memory_used_bytesbytes
h2ogpte_gpu_memory_total_bytesbytes
h2ogpte_gpu_temperature_celsius°C

Values are read with nvidia-smi and cached for 10 seconds. On a pod without a GPU, or when nvidia-smi fails, the GPU series are omitted and the rest of the endpoint still works.

If you run the Prometheus Operator (for example through kube-prometheus-stack), enable the ServiceMonitor in the Helm chart:

models:
  serviceMonitor:
    enabled: true
    interval: 30s
    # Match the serviceMonitorSelector of your Prometheus, if it has one.
    labels:
      release: kube-prometheus-stack

It scrapes the models pods and, when modelsHeavy.enabled is set, the heavy GPU tier too. This gives a per-pod view: which models pod is using which GPU, and how much.

Node-level metrics with DCGM​

For the full set of NVIDIA DCGM metrics (SM clocks, power draw, XID errors, NVLink bandwidth and more), use the dcgm-exporter that ships with the NVIDIA GPU Operator. The GPU Operator Helm chart deploys it through dcgmExporter.enabled and can create its own ServiceMonitor with dcgmExporter.serviceMonitor.enabled=true (check the default of your operator version with helm show values nvidia/gpu-operator). See NVIDIA's guide Integrating GPU Telemetry into Kubernetes and the dcgm-exporter project for details. Metrics such as DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_FB_USED are then available in the same Prometheus.

The h2oGPTe Helm chart does not deploy dcgm-exporter itself.

Feedback