提出统一框架，用放置语义解析分布式训练的内存与通信开销。
Placement Semantics for Distributed Deep Learning: A Systematic Framework for Analyzing Parallelism Strategies
- 用四种状态在五种模式下的放置方式描述并行策略。
- 准确预测内存占用和通信量，与论文结果完全一致。
- 可安全组合多种策略，适合大规模模型训练设计者。
训练大语言模型需在多个加速器间分配计算，但从业者常通过试错选择并行策略（如数据并行、张量并行、流水线并行、ZeRO），因缺乏统一系统框架预测其行为。本文提出放置语义：每种策略由参数、优化器、梯度、激活这四个训练状态在设备间的分布方式决定，采用复制、分片、分片+收集、实例化、卸载五种模式。仅凭放置信息即可推导出内存消耗和通信量。预测结果与已有论文完全一致：ZeRO-3 在 1.5 倍通信成本下比数据并行节省 8 倍内存。我们证明了梯度完整性与状态一致性是分布式训练等价于单设备的充要条件，并给出安全组合策略的规则。该框架统一了 ZeRO Stage 1–3、FSDP、张量并行和流水线并行，均为不同放置选择的实例。
原文摘要 · Abstract (English)
Training large language models requires distributing computation across many accelerators, yet practitioners select parallelism strategies (data, tensor, pipeline, ZeRO) through trial and error because no unified systematic framework predicts their behavior. We introduce placement semantics: each strategy is specified by how it places four training states (parameters, optimizer, gradients, activations) across devices using five modes (replicated, sharded, sharded-with-gather, materialized, offloaded). From placement alone, without implementation details, we derive memory consumption and communication volume. Our predictions match published results exactly: ZeRO-3 uses 8x less memory than data parallelism at 1.5x communication cost, as reported in the original paper. We prove two conditions (gradient integrity, state consistency) are necessary and sufficient for distributed training to match single-device results, and provide composition rules for combining strategies safely. The framework unifies ZeRO Stages 1-3, Fully Sharded Data Parallel (FSDP), tensor parallelism, and pipeline parallelism as instances with different placement choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。