When One GPU Stops the Entire Training Job

Meta trained Llama 3 405B using up to 16,384 NVIDIA H100 GPUs. During a 54-day snapshot of pre-training, it recorded 466 job interruptions: 47 planned and 419 unexpected. Around 78% of the unexpected interruptions involved confirmed or suspected hardware problems. The paper attributes 58.7% of unexpected interruptions to GPU issues.

The paper also reports above 90% effective training time, meaning useful training time divided by elapsed time. During the 54-day snapshot, only three incidents required significant manual intervention. Source: The Llama 3 Herd of Models.

That is a useful combination to think about. A system can experience frequent interruptions and still spend most of its time making progress, provided the path from failure to recovery is fast and dependable.

The annoying failures do not always crash, either. A process can remain alive while the operation everyone is waiting for has stopped.

To understand the machinery this requires, let us follow a failure through a small synchronous training job. The coordinator and recovery protocol below are a simplified design for a Go lab called training-goodput-lab.

01The next step needs every participant

Meta explicitly connects the restart problem to synchronous training: a single GPU failure may require restarting the entire job. Source.

For the small model, imagine eight workers. Each performs its assigned work for a training step and reports completion to a coordinator. The coordinator advances the global step only after all eight finish.

Seven completion messages are insufficient. The eighth worker's contribution is required.

Real distributed training uses collective communication and several forms of parallelism. Our barrier collapses that complexity into one rule: every required participant must reach the step boundary before the job advances.

Suppose seven workers finish their local work in 100 milliseconds and one takes 500 milliseconds. Ignoring communication overhead, the step takes at least 500 milliseconds. If the eighth worker never finishes, the job cannot reach the next step at all.

Adding more healthy workers does not produce the missing contribution. Replacing the worker also requires care: its replacement must join with the state the rest of the job expects.

This makes recovery a coordinated operation. The system needs to decide which attempt is over, which state remains usable, and when all participants are ready to continue.

02A heartbeat can arrive after training stops

A worker can fail in several ways, and each leaves a different signal.

A heartbeat tells us that something can still send a heartbeat. If the heartbeat loop runs independently of the training operation, it can remain perfectly healthy while training is stuck.

The paper describes remote-GPU or NVLink failures that stall CUDA load/store operations without producing a clear error code. Source.

In the lab, I would model that with a worker that keeps its TCP connection open and continues sending heartbeats, but never completes its current step.

A coordinator watching only heartbeats has no reason to declare it unhealthy. A separate progress watchdog does: it knows which step should be running, when that attempt started, and which workers have finished their local work.

The watchdog also needs to understand the current phase. A deliberate checkpoint pause should not look like a hung training step. Startup, checkpointing, restoration, and training need their own deadlines.

Stragglers need another measurement. Record each worker's local completion time before it waits at the barrier. If every worker's timer ends when the global barrier opens, the healthy workers inherit the slow worker's wait, and everyone appears equally slow.

For identical simulated workers, a sliding window and a configurable multiple of the median are enough to demonstrate the idea. Requiring several slow steps reduces the chance of replacing a worker because of one noisy measurement.

Replacement has a cost, though. A small temporary slowdown may be cheaper than a rollback and restart. The useful decision is whether the expected future delay exceeds the recovery cost.

There is also a limit to what a timeout proves. It establishes that expected progress did not arrive. In a real collective, identifying the GPU, link, or peer that caused the stall requires more evidence than the timeout itself.

03Even the afternoon can change the baseline

Meta observed a 1–2% daily throughput variation associated with higher midday temperatures affecting GPU voltage and frequency scaling. Source.

That is a small detail with a useful consequence for the detector.

A relative straggler check asks whether one worker is unusually slow compared with its peers. If the whole cluster slows down together, the median moves too. Every worker can look normal relative to the rest while the job's throughput declines.

I would model that separately from a broken worker. A gentle shared drift in step duration can represent changing operating conditions; a large persistent delay on one worker can represent a local straggler.

The appropriate response differs. Replacing workers one after another during a shared slowdown can add recovery overhead without addressing the cause.

For this experiment, I would watch both per-worker deviation and the overall step-duration baseline. The first helps locate an outlier. The second reveals that the job itself has become slower.

A timeout set just above yesterday's normal step time is fragile. A detector needs enough context to distinguish expected variation, sustained degradation, and an operation that has stopped progressing.

04Recovery needs one consistent state

Take a concrete example:

The job discards 47 completed steps, plus the partial work spent attempting step 1,048.

That distinction matters in the metrics. The incomplete step never became global progress. The other 47 did, but recovery subsequently removed them.

For this design, recovery would first stop accepting completions from the failed attempt. It would remove the suspected worker from active membership, promote a spare into that logical position, and restore every participant from the same completed checkpoint. Training would resume only after all required workers acknowledged restoration.

A recovery generation number makes this safer. Every command and completion carries the generation it belongs to. When recovery starts, the coordinator advances that number.

A delayed message from the previous generation then remains identifiable even if it arrives after the replacement has connected. It cannot accidentally complete a step in the new attempt.

The checkpoint must contain enough state to resume the computation. PyTorch's checkpoint guidance explicitly includes optimizer state when resuming training, because model parameters alone do not capture everything being updated. PyTorch checkpoint documentation.

For the lab, that state can stay small: a step counter, a deterministic fake loss, and a byte blob. If the computation uses a random stream, its position also belongs in the saved state.

The test is straightforward: save, advance, fail, restore, and compare the restored state with the checkpoint. A process printing “resumed” is only the beginning of that verification.

Storage ownership matters too. A spare needs access to the failed worker's checkpoint shard. A shared directory can demonstrate this on one machine, but files that survive a process kill do not demonstrate recovery from losing the machine and its disk.

05Checkpoint frequency decides how much work gets repeated

A checkpoint limits how far the job needs to roll back. Producing it also takes time.

Save too frequently and training spends a large fraction of its runtime creating checkpoints. Leave long gaps and each interruption can discard more computation.

Start with a deliberately simple model:

• C is the time spent blocking training to write a checkpoint.
• T is useful computation time between checkpoint writes.
• M is the mean interval between interruptions of the whole job.

When interruptions are approximately random and rare within an interval, a failure loses about half an interval of computation on average. The first-order checkpoint and replay overhead is:

W(T) ≈ C / T + T / (2M)

The first term decreases as checkpoints become less frequent. The second increases. Minimizing their sum gives the familiar Young first-order estimate:

T* ≈ sqrt(2 × C × M)

This is an approximation for blocking checkpoints with small overheads. Research on checkpoint intervals makes those assumptions explicit; Daly's work develops more complete models.

The interval here is computation between writes. Add the checkpoint duration if you want the full compute-and-save cycle.

Now give the formula some numbers. The snapshot's observed average spacing between unexpected interruptions is:

54 × 24 / 419 ≈ 3.09 hours

For an illustrative calculation, use that spacing as M and assume a blocking checkpoint costs 30 seconds:

T* ≈ sqrt(2 × 30 × 11,135)
   ≈ 817 seconds
   ≈ 13.6 minutes

The 30-second cost is an invented input for this example. The resulting interval is an analytical estimate, not a setting reported by Meta or a measured result from the lab.

Using the same approximation:

These values exclude detection and recovery downtime. They illustrate the tradeoff, rather than predict the total efficiency of a real training run.

There is an easy plotting mistake here: overhead forms the U-shaped curve. Goodput should rise toward a maximum and then fall.

The formula also produces a time interval. A policy expressed as “checkpoint every K steps” needs a step duration before the two can be compared. As step times change, the same K represents a different amount of exposed work.

Finally, the three-hour figure describes job interruptions. The count includes several kinds of failures. Converting it into a per-worker rate for a simulator requires an explicit assumption about how interruptions are distributed across workers.

06Async checkpoints have two completion times

Blocking checkpoints put the whole save on the training path. An asynchronous design can capture a stable snapshot, resume training, and finish writing the snapshot in the background.

PyTorch's explanation separates copying state into CPU memory from persisting it to storage. Training can resume while the latter continues. PyTorch on asynchronous checkpointing.

That creates two milestones:

1. The captured state is safe from subsequent training updates.
2. Every required part of the checkpoint has been saved and the checkpoint is recoverable.

Those milestones can be far apart.

Suppose step 1,000 is the latest completed checkpoint. A new snapshot is taken at step 1,050, and training continues. A failure occurs before the new write finishes.

Under a policy that abandons incomplete saves on recovery, the job still restores step 1,000. Starting a newer checkpoint did not make it available.

The proposed checkpoint protocol assigns each save a unique identifier and publishes a manifest only after all required worker files are complete. The manifest would name the logical step, worker shards, and checksums. Until then, the previous manifest would remain the recovery target.

The captured state also needs to stay immutable. Passing a pointer to a live mutable blob into a background writer can produce a file assembled from different moments in training. A stable copy or equivalent isolation is part of checkpoint correctness.

Old background operations need fencing as well. A save started before recovery must not later publish itself as the latest checkpoint for the new attempt.

Async checkpointing introduces memory pressure and overlapping I/O. PyTorch's recipe recommends limiting concurrent checkpoint requests for most users. Async checkpoint recipe.

A small lab can start with one write in flight. If another checkpoint becomes due, the policy must explicitly wait, skip, or reschedule it. An unlimited queue of snapshots would hide a growing problem behind a low snapshot-pause metric.

07Measure the work that survives

The lab needs a precise definition of goodput.

Counting every completed step is insufficient because some completed steps will later be discarded. Counting all time spent inside the simulated training function would also reward repeated computation.

For a run beginning at step zero, with a fixed ideal duration per logical step, I would use:

useful work = retained steps × ideal step duration
goodput = useful work / elapsed wall time

The final checkpoint must actually finish, and its completion time belongs in the denominator. Otherwise, the experiment gives itself a free save at the end.

Suppose the final checkpoint preserves 3,600 steps, each representing two seconds of ideal work. If the run takes 9,000 seconds, goodput is 80%.

The numerator contains 7,200 seconds of retained work. Time spent replaying discarded steps, waiting on stragglers, detecting faults, restoring state, and pausing for checkpoints remains in the elapsed time.

For this fixed-cost model, jitter and throttling would add delays to the ideal step duration. That keeps the ratio at or below one without clamping a broken calculation.

This normalized progress metric also gives a slowdown somewhere to appear. A worker taking twice as long to perform the same step does not produce twice as much useful work.

Useful companion metrics are current step, latest recoverable step, discarded steps, fault-to-detection time, restore time, and time until the job advances beyond its previous progress.

That last measurement is useful. Workers can resume quickly and then spend a long time replaying lost steps before making new progress.

Background checkpoint write duration deserves its own metric. Because it overlaps training, it cannot simply be added to blocking time as though the two consumed separate portions of the clock.

The lab's definition should remain visible beside its results. Meta's effective-training-time number uses its own accounting, and hardware utilization measures another aspect of the system.

08A Go lab for exercising the recovery path

The project I would build around these ideas is training-goodput-lab: a coordinator, simulated workers, checkpoint storage, a fault injector, and a small set of measurements.

The design has two modes that exercise the same coordinator decisions.

Live mode would use real Go processes and TCP connections. Workers would write and restore actual files. Killing a worker, hanging its step loop while preserving heartbeats, or slowing its work would exercise the detection and recovery path.

Eight workers and a couple of spares would be enough for this mode. The goal would be to observe the exact state transitions and inspect what reached disk.

Simulation mode would use a virtual clock to explore larger worker counts and many simulated days. It would advance between meaningful events without waiting for wall-clock sleeps. The coordinator's membership, checkpoint, and recovery rules would remain shared with live mode.

The simulator must also declare what happens when spares run out: workers return after repair, replacements come from a finite budget, or the run stops. Two permanently consumed spares cannot support hundreds of interruptions.

Its failure preset could match the observed job interruption frequency. Distributing that frequency across workers as independent random events would be a modeling choice. The crash, hang, and slowdown mixture would be another chosen input.

I would begin with four experiments:

A heartbeat-only hang may remain undetected for the entire observation window. That result should be recorded as “not detected within the run.” Assigning it an invented finite latency would make the comparison misleading.

For policy comparisons, I would replay the same generated fault trace. Reusing a seed alone is insufficient if different policies consume random numbers in different orders. A checkpoint decision should not silently change which failure happens next.

The scale sweep also needs a narrow interpretation. Holding the ideal step cost fixed isolates reliability effects as worker count rises. It does not model real communication scaling or predict production training throughput.

09The failure drill I would start with

Start with a worker that remains connected and sends heartbeats while refusing to complete its step.

Watch the progress deadline expire. Confirm that the coordinator ends the current attempt, selects the latest completed checkpoint, and restores every participant to the expected state.

Then follow the job past restoration. Check when it resumes, how much work it repeats, and when it finally moves beyond the progress it had made before the fault.

Those timestamps explain where the runtime went. A slow detector calls for different work than a slow restore. A large rollback calls attention to checkpoint spacing or checkpoint completion lag. Repeated unnecessary replacements point back to the detector.

The useful outcome is a recovery path whose cost is visible and whose state transitions are testable. Once that is in place, the checkpoint sweep has something trustworthy to measure.