Meta has revealed that more than 11 percent of the GPUs at its Prometheus AI supercluster in New Albany, Ohio, are not actively training models at any given moment. Some are dead, but most are hot spares: each 72-GPU GB200/GB300 NVLink domain runs only 64 GPUs, leaving eight ready to take over if a GPU fails. The redundancy is designed to stop a single failure from taking out an entire rack and forcing a training run to roll back.
The company says this approach helped cut training interruptions from roughly 1,000 per day for every 8,000 GPUs in the first phase to about 11 per day, a 99 percent reduction. Meta also uses data parallelism to isolate faults to a single model replica, and a distributed checkpointing scheme that saves model state to multi-tier storage. A full interruption now happens only when a fault exceeds the data-parallel domain.
The hardest challenge was scaling NVLink fabrics. With more than 600,000 NVLink connections, even marginal signal degradation triggers frequent recovery cycles, so Meta runs continuous telemetry and L11 diagnostic burns to find weak links before production. Power transients from ramping tens of thousands of GPUs are another concern. Meta reports more than 85 percent goodput for pretraining and over 90 percent for reinforcement learning on its Muse Wonder model.
Prometheus is expected to scale to about a million accelerators and a gigawatt of capacity. Meta's next target is an interruption rate of just two per day per 8,000 GPUs, which would amount to roughly 100,000 GPU-hours between interrupts.