用自编码器压缩模型分片通信，让边缘设备高效微调大模型。
AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning
- 在模型分片处加轻量自编码器压缩中间特征与梯度。
- 两阶段对齐机制解决预训练模型与压缩器的特征分布不匹配问题。
- 适合资源受限的边缘设备上大模型微调，通信量大幅降低。
大规模基础模型（FMs）在资源受限的边缘设备上进行分布式微调（DFT）受限于本地计算能力与通信开销。并行分片学习（PSL）通过将少量模型层保留在客户端，其余计算卸载至服务器，降低了客户端计算负担；但每个训练步仍需客户端与服务器交换中间激活值和梯度。现有通信压缩方法多依赖任务无关启发式策略，如稀疏化与量化。可学习的压缩器虽能更好适应中间表示，但需与目标模型共同训练，直接嵌入现成预训练模型会导致特征分布错位，损害微调性能。为此，我们提出AE-PSL，一种基于轻量级自编码器（AE）在分片层压缩中间激活与梯度的通信高效框架。为确保压缩器与预训练模型兼容，AE-PSL引入新颖的两阶段对齐机制，在微调前分别适配预训练模型的特征流形与客户端特定的特征分布。
原文摘要 · Abstract (English)
Distributed Fine-Tuning (DFT) of large-scale Foundation Models (FMs) on resource-constrained edge devices is limited by local compute constraints and communication overhead. Parallel Split Learning (PSL) reduces client-side computation by keeping few model layers on each client and offloading the remaining computation to the server; however, clients must exchange intermediate activations and gradients with the server at every training step. Existing SL communication-compression methods mainly rely on task-agnostic heuristics, such as sparsification and quantization. While learnable SL compressors can better adapt to intermediate representations, they require co-training with the target model. Therefore, directly inserting them into off-the-shelf FMs introduces feature-distribution misalignment and degrades DFT performance. To address this, we propose AE-PSL, a communication-efficient PSL framework that compresses intermediate activations and gradients using a lightweight AutoEncoder (AE) placed at the split layer. To ensure compatibility of AE compression with pre-trained FMs, AE-PSL introduces a novel two-stage alignment mechanism, which adapts the AE to the pre-trained model's feature manifold and client-specific feature distributions before DFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。