Guide
Which GPU signals matter, and what they tell you
A practical guide to GPU telemetry for teams running training and inference.
By the C-Gen.AI engineering team
Utilization is not one number
The GPU utilization most tools report is the share of time at least one kernel was running. A GPU can read 100% while a small kernel keeps only part of the chip busy. Metrics that look inside the chip, such as SM activity and occupancy, give a truer picture of how much work is being done.
Health signals worth watching
Standard GPU health telemetry covers ECC memory errors, PCIe errors, XID events from the driver, temperature and power excursions, and interconnect errors. Tracking them lets you take a failing GPU out of service instead of discovering the problem from a crashed job.
Usage answers a different question
Telemetry says how busy a GPU is. Usage says who used it: which project, team and user consumed which GPU hours. Engineers need the first to tune jobs; team leads and finance need the second to plan budgets and charge back costs. Slurm can record job accounting, but turning those records into reports usually takes extra work.
Put signals next to workloads
A temperature spike means little on its own. Seen next to the training job or endpoint running on that GPU, it tells you what’s at risk, who owns it and whether to act now or at the next checkpoint.
