Cloudfoxy Labs

← All labs

Lab 07: GPU Operator, dcgm-exporter, Prometheus and Grafana

Level 3 (Operate) · GPU tier

180 minutes suggested, up to 360 in one session. Price: 5 credits per minute. Includes Check my work.

Hosted, this lab covers Parts A, B, C, D, E. The other Parts need your own machine (Home Lab Kit) and are shown collapsed in the lab.

The GPU tier is not available on this site yet.

What you will be able to do

  • Name every GPU Operator component on a node and say what each one does.
  • Read dcgm-exporter metrics, and tell the legacy metric names from the DCGM 4.6+ canonical names.
  • Install kube-prometheus-stack with Helm and make Prometheus scrape dcgm-exporter with a ServiceMonitor you wrote.
  • Import NVIDIA's DCGM dashboard into Grafana and read its panels under load.
  • Write PromQL alert rules for GPU temperature, XIDs and a missing exporter, and prove them with promtool test rules.
  • Find out why a GPU metric or alert has gone silent.