Meta leaves over 11 percent of its flagship AI cluster idle to prevent training crashes

Meta has disclosed rare operational details about Prometheus, its massive AI supercluster in New Albany, Ohio, which currently trains its Muse model family and is scaling toward roughly 1 million Nvidia accelerators and one gigawatt of power. To keep large-scale training runs from halting, Meta leaves more than 11 percent of the system's GPUs idle or offline as hot spares. Out of every 72 GPUs across its Nvidia GB200 and GB300 NVL72 rack architectures, only 64 actively run workloads, while the remaining eight stand ready to replace hardware failures immediately. [1][2]

The operational buffer addresses the immense fragility of hyperscale AI deployments, where single-component failures would otherwise take down entire racks. Meta says it endured about 1,000 interruptions daily for every 8,000 GPUs during Prometheus's initial phase. Alongside hot-spare swapping, Meta introduced data-parallel recovery to isolate failing model replicas, distributed asynchronous checkpointing, and diagnostic burns to stabilize more than 600,000 NVLink connections. These measures slashed interruptions by 99 percent to around 11 daily per 8,000 GPUs, allowing pretraining on its Muse Wonder model to reach over 85 percent goodput. [1][2]

Meta aims to eventually drive down interruptions to just two per day for every 8,000 GPUs, or approximately 100,000 GPU hours between incidents. The company is preparing to add Nvidia Vera Rubin systems to Prometheus and is applying its architectural findings to Hyperion, a next-generation supercluster under development in Richland Parish, Louisiana. [1][2]