动态冻结参数，提升流水线并行训练速度40%
TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism
- 将流水线调度建模为有向无环图，求解线性规划确定最优冻结比例
- 在保持精度的前提下，使LLaMA-8B训练吞吐量提升最高达40%
- 适用于多种流水线并行场景，适合大规模模型训练者
流水线并行可突破单设备内存限制训练大模型，但实际吞吐受限于流水线空泡。虽然参数冻结可通过自适应跳过反向计算提升吞吐，现有方法常过度冻结，导致精度下降。为此，我们提出TimelyFreeze，将流水线调度建模为有向无环图，并求解线性规划以计算在精度约束下最小化批次执行时间的最优冻结比例。实验表明，TimelyFreeze在LLaMA-8B上实现最高达40%的训练吞吐提升，且精度相当。整体上，该方法在不牺牲收敛性前提下加速大规模模型训练，并可泛化至多种流水线并行设置。
原文摘要 · Abstract (English)
Pipeline parallelism enables training models that exceed single-device memory, but practical throughput remains limited by pipeline bubbles. Although parameter freezing can improve training throughput by adaptively skipping backward computation, existing methods often over-freeze parameters, resulting in unnecessary accuracy degradation. To address this issue, we propose TimelyFreeze, which models the pipeline schedule as a directed acyclic graph and solves a linear program to compute optimal freeze ratios that minimize batch execution time under accuracy constraints. Experiments show that TimelyFreeze achieves up to 40% training throughput improvement on LLaMA-8B with comparable accuracy. Overall, it enables faster large-scale model training without compromising convergence and generalizes across diverse pipeline-parallel settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。