CollaPipe让手机和边缘服务器协作训练大模型,效率提升超一半。
CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks
- 把大模型分段部署在手机和边缘服务器,动态分配计算任务。
- 相比传统方法,计算效率最高提升15.09%,端到端延迟降低48.98%。
- 适合资源不均的移动边缘网络,支持在线学习与自适应调整。
智能移动应用对基于Transformer的大语言模型(LLMs)多智能体协作的需求日益增长,但其在移动边缘计算(MEC)网络中的训练仍面临计算负载重、端到端延迟高和模型泛化能力弱等挑战。本文提出CollaPipe,一种融合协同流水线并行与联邦聚合的混合分布式学习框架,支持自演化智能网络。其中,编码器部分被自适应划分为可变大小的片段,分布于移动设备上进行流水线并行训练,解码器则部署于边缘服务器处理生成任务,并通过联邦聚合实现全局模型更新。为提升训练效率,我们构建联合优化问题,自适应分配模型片段、微批次、带宽及传输功率。基于李雅普诺夫优化推导闭式收敛界,设计了动态分段调度与资源分配(DSSDA)算法,确保长期约束下的系统稳定性。在下游任务上使用Transformer和BERT模型的大量实验表明,CollaPipe可将计算效率提升最高达15.09%,端到端延迟至少降低48.98%,单设备内存占用减少超过一半,成功实现在异构动态通信环境中的在线学习。
原文摘要 · Abstract (English)
The increasing demand for intelligent mobile applications has made multi-agent collaboration with Transformer-based large language models (LLMs) essential in mobile edge computing (MEC) networks. However, training LLMs in such environments remains challenging due to heavy computation, high end-to-end latency, and limited model generalization. We introduce CollaPipe, a hybrid distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving intelligent networks. In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks. Then we perform global model update via federated aggregation. To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power. We derive and use a closed-form convergence bound to design an Dynamic Segment Scheduling and Resource Allocation (DSSDA) algorithm based on Lyapunov optimization, ensuring system stability under long-term constraints. Extensive experiments on downstream tasks with Transformer and BERT models show that CollaPipe improves computation efficiency by up to 15.09%, reduces end-to-end latency by at least 48.98%, and cuts single device memory usage by more than half, enabling online learning in heterogeneous and dynamic communication environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。