让大模型联邦学习通信与计算并行，提速降延迟。
ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
- 通信与计算分线并行，减少等待时间。
- 在指令遵循和数学推理任务上保持性能，通信效率提升明显。
- 适合资源受限设备参与大模型联邦训练。
联邦学习（FL）作为隐私保护的训练范式受到广泛关注。近年来，联邦块坐标下降方法因其允许客户端仅本地训练模型的子集而成为大规模模型训练的热门选择。然而，在大语言模型（LLMs）时代，即使单个块也包含大量参数，导致通信延迟显著，尤其对资源受限的客户端影响更大。为应对大模型联邦训练/微调中的这一挑战，本文提出ParaBlock，通过建立通信与计算的双并行线程，提升通信效率。理论上证明了ParaBlock的收敛速度与标准联邦块坐标下降方法一致。在通用指令遵循和数学推理任务上的微调实验表明，ParaBlock不仅保持了强性能，还显著提升了通信效率。
原文摘要 · Abstract (English)
Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。