用小MLP层替代LoRA，提升大模型联邦微调效率
Sequential Compression Layers for Efficient Federated Learning in Foundational Models
- 在Transformer的前馈网络中插入小型MLP层作为可训练模块
- 在语言和视觉模型上均优于现有基于LoRA的方法
- 适合资源受限环境下高效联邦微调大型模型
联邦学习（FL）在多个节点间使用私有数据微调大语言模型（LLMs）方面日益流行。尽管LoRA已被广泛用于参数高效的联邦微调，但近期的理论与实证研究指出其在联邦学习场景下表现不佳。为此，我们提出一种新颖、简单且更有效的参数高效微调方法，不依赖于LoRA。该方法在Transformer块的前馈网络中，于原有的up proj（自注意力模块后的FFN投影层）与down proj之间引入一个小型多层感知机（MLP）层。此方案解决了LoRA在联邦微调中的瓶颈问题，并在语言模型和视觉编码器上均展现出优于近期基于LoRA方法的性能。
原文摘要 · Abstract (English)
Federated Learning (FL) has gained popularity for fine-tuning large language models (LLMs) across multiple nodes, each with its own private data. While LoRA has been widely adopted for parameter efficient federated fine-tuning, recent theoretical and empirical studies highlight its suboptimal performance in the federated learning context. In response, we propose a novel, simple, and more effective parameter-efficient fine-tuning method that does not rely on LoRA. Our approach introduces a small multi-layer perceptron (MLP) layer between two existing MLP layers the up proj (the FFN projection layer following the self-attention module) and down proj within the feed forward network of the transformer block. This solution addresses the bottlenecks associated with LoRA in federated fine tuning and outperforms recent LoRA-based approaches, demonstrating superior performance for both language models and vision encoders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。