Lab 12: NCCL tests and Magnum IO debugging
150 minutes suggested, up to 300 in one session. Price: 5 credits per minute.
Hosted, this lab covers Parts A, B, C, D, E. The other Parts need your own machine (Home Lab Kit) and are shown collapsed in the lab.
The GPU tier is not available on this site yet.
What you will be able to do
- Build NVIDIA's
nccl-testsagainst the CUDA toolkit and NCCL from NVIDIA's repository. - Run
all_reduce_perfand read every column, including why algbw and busbw differ. - Turn NCCL's logging on and off with
NCCL_DEBUG, narrow it withNCCL_DEBUG_SUBSYS, and send it to per-process files. - Find and remove NCCL settings that do not belong on a machine, including ones hidden in configuration files.
- Follow a multi-node investigation on InfiniBand:
srun --mpi=pmix,ibstat,iblinkinfo -l,perfqueryandib_write_bw. - Check whether GPUDirect RDMA is in use, and which kernel path (DMA-BUF or
nvidia-peermem) provides it. - Read an evidence bundle from a slow multi-node test and name the node, the layer and the fix.