Frontier training runs are long-haul operations: thousands of GPUs, weeks of runtime, and a budget that measures failures in days, not dollars. The lessons below are the ones that survive contact with the data center.
1. Checkpoints are the product
A training run is a process that occasionally produces checkpoints. Everything — fabric topology, job scheduler, storage tiering — exists to make checkpoint save and restart fast and correct.
2. Nodes die on weekends
Not statistically, but emotionally. Fault-tolerant jobs aren't a feature; they're a staffing strategy. When the dead node is detected and rescheduled without a human, the weekend stays boring.
3. The fabric is the load-bearing wall
You can replace a GPU. You cannot easily replace a slow rail. Monitor link errors and congestion before utilization — utilization is a symptom, fabric health is the cause.
4. Log everything, alert on almost nothing
Telemetry is for forensics; alerts are for the three conditions that actually wake a human: job death, data loss risk, and fabric degradation. Everything else can wait for morning.
5–12. The unglamorous ones
Pin configuration drift, thermal asymmetry between racks, NTP, the one switch firmware version nobody updated, disk-full on the log partition, and the fact that "it worked on 64 GPUs" predicts nothing at 1,000. Write them all down so the next run doesn't have to learn them again.