Cloudfoxy Labs

← All labs

Lab 08: nvidia-smi, DCGM diagnostics and health, NVSM

Level 3 (Operate) · GPU tier

150 minutes suggested, up to 300 in one session. Price: 5 credits per minute. Includes Check my work.

Hosted, this lab covers Parts A, B, C, D, E, F. The other Parts need your own machine (Home Lab Kit) and are shown collapsed in the lab.

The GPU tier is not available on this site yet.

What you will be able to do

  • Read any section of nvidia-smi -q, write CSV queries for scripts, and explain why a GPU is running below its clocks.
  • Install DCGM, and use discovery, groups, dmon, health watches and job statistics.
  • Choose and run the right dcgmi diag level, and read its result.
  • Set DCGM policies in both the legacy and the DCGM 4.7 field-based form.
  • Classify an XID from a kernel log line and decide between restart, reset, reboot and RMA.
  • Build a health check that drains a Slurm node with a useful reason, and return the node to service the right way.
  • Read NVSM's health output on a DGX and say what it covers that DCGM does not.