Every few months, support queues get the same ticket: "Training is 40% slower than last week, nothing changed." Usually, something did change — and usually it lives in memory, not in compute.
Start from the memory wall
Modern accelerators compute faster than they can feed themselves. The gap is why HBM exists and why its bandwidth, not its capacity, is the spec that decides throughput for most workloads. A model that streams activations will stall on a memory-bound kernel no matter how many TFLOPS are on the box.
Read the error counters early
HBM cells fail slowly and predictably. Correctable errors climb for weeks before a row dies. Good fleet telemetry tracks corrected-error rates per stack, per day — a slow climb is a maintenance event, a spike is a node swap. Teams that watch these numbers replace hardware before they lose a checkpoint.
More bandwidth beats more GPUs — sometimes
If your run is memory-bound and your model fits on one node, adding nodes adds communication cost, not throughput. An H200's 141 GB at 4.8 TB/s can beat two smaller cards in the same chassis for mid-size workloads — and a B200's 192 GB at 8 TB/s changes the calculus again. Publish the numbers and the honest answer is workload-dependent — but the methodology is stable: measure MFU per dollar, not GPUs per headline.
The checklist
When a run slows down, check in this order: HBM error counters, per-GPU clock throttling, fabric link errors, then your own code. In practice, hardware tells the truth first.