Operational Analysis of LLM Pre-training on 504-GPU Cluster Reveals Failure Patterns
A new technical report from arXiv presents an empirical analysis of large-scale AI training operations on a 63-node NVIDIA B200 cluster comprising 504 GPUs. The study, conducted by a consortium including SKT, Upstage, Lablup, NVIDIA Korea, and VAST Data, utilizes 55 days of Prometheus metrics and 73 days of operational logs from 224 multi-node training sessions. Key findings highlight that hardware failures are routine in production environments. The team achieved a 100% detection rate for GPU failures using a multi-signal strategy, though no single metric was universally dominant. Researchers identified a 'bandwidth paradox' where low RoCE utilization was caused by NFS RPC layer saturation rather than network limits. Additionally, automated recovery mechanisms proved significantly more effective than manual interventions, with a 33.3% success rate compared to 12.5% for manual retries. The cross-organizational collaboration enabled the diagnosis of complex storage I/O bottlenecks that only manifest at scale, offering valuable insights for optimizing distributed systems in large language model pre-training.
Wire timeline
Operational Analysis of LLM Pre-training on 504-GPU Cluster Reveals Failure Patterns
A new technical report from arXiv presents an empirical analysis of large-scale AI training operations on a 63-node NVIDIA B200 cluster comprising 504 GPUs. The study, conducted by a consortium including SKT, Upstage, Lablup, NVIDIA Korea, and VAST Data, utilizes 55 days of Prometheus metrics and 73 days of operational logs from 224 multi-node training sessions. Key findings highlight that hardware failures are routine in production environments. The team achieved a 100% detection rate for GPU failures using a multi-signal strategy, though no single metric was universally dominant. Researchers identified a 'bandwidth paradox' where low RoCE utilization was caused by NFS RPC layer saturation rather than network limits. Additionally, automated recovery mechanisms proved significantly more effective than manual interventions, with a 33.3% success rate compared to 12.5% for manual retries. The cross-organizational collaboration enabled the diagnosis of complex storage I/O bottlenecks that only manifest at scale, offering valuable insights for optimizing distributed systems in large language model pre-training.
cs.AI updates on arXiv.org