用轻量模型预测大规模分布式训练性能，省去试错成本
ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork
- 基于模型特征和硬件拓扑，解析式预测跨集群训练耗时
- 发现岛屿规模增大后性能增益递减，明确计算与通信瓶颈切换点
- 适合想快速选最优部署方案的团队，尤其大型模型训练场景
训练大规模AI模型常需跨数据中心的分片、多集群、去中心化训练。但资源分配空间巨大，穷举测试与手动调优不可行，性能受模型大小、显存、批处理量、带宽及分片策略等紧密耦合因素影响。我们提出ShardMeter，一种轻量级分析性能模型，可预测任意分片、分布式甚至去中心化训练中Transformer类任务的端到端运行时间。给定模型特性和目标硬件拓扑，ShardMeter能估算每卡与每岛吞吐率、训练成本、总时钟时间，并定位性能瓶颈。分析揭示岛屿规模增大进入收益递减区，量化计算与通信主导的扩展转换点，评估超参数权衡，并建模大规模去中心化训练的成本-吞吐关系。ShardMeter使用户快速探索配置空间，选择近优部署方案，避免高成本试错。
原文摘要 · Abstract (English)
Training large-scale AI models often outgrows a single data center, demanding sharded, multi-cluster, and decentralized training. However, the huge space of resource allocations makes exhaustive benchmarking and manual tuning impractical, while performance depends on tightly coupled factors like model size, GPU memory, batch size, bandwidth, and sharding strategy. We introduce ShardMeter, a lightweight analytical performance model that predicts the end-to-end runtime of transformer-based workloads across arbitrary sharded, distributed, and even decentralized training. Given a model's characteristics and a target hardware topology, ShardMeter estimates per-GPU and per-island throughput, training cost, total wall-clock time, and identifies performance bottlenecks. Our analysis reveals diminishing-return regimes as island size increases, quantifies transitions between compute- and communication-bound scaling, evaluates hyperparameter trade-offs, and models cost-throughput for large-scale decentralized training. ShardMeter exposes these insights to quickly explore the configuration space, choose near-optimal deployment plans, and avoid costly trial and error.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。