用稀疏适配器实现高效模型合并，性能优于LoRA和全量微调。
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
- 仅训练基础网络中部分权重，构造稀疏适配器，方法更简洁。
- 合并20个NLP任务适配器后，稀疏适配器在分布内任务上表现最佳。
- 适合需要快速构建可扩展模块化模型的研究者使用。
参数高效任务专家的合并近年来受到广泛关注，有助于构建可动态适应特定下游任务的模块化架构，而无需额外微调。通常以LoRA作为此类架构的基础，利用低秩权重结构减少可训练参数。本文研究稀疏适配器的性质，即仅训练基础神经网络中的部分权重，作为模块化架构的潜在构建单元。首先，提出一种简单有效的稀疏适配器训练方法，概念上比现有方法更简洁，且在实验中出人意料地优于LoRA和全量微调。其次，通过合并最多20个自然语言处理任务的适配器，探索其可扩展性，显著超越文献中常见的规模。结果表明，稀疏适配器在合并后表现出更优的分布内性能，优于LoRA或全模型合并。然而，所有方法在未见任务上的保持性能仍具挑战。
原文摘要 · Abstract (English)
Merging parameter-efficient task experts has recently gained growing attention as a way to build modular architectures that can be rapidly adapted on the fly for specific downstream tasks, without requiring additional fine-tuning. Typically, LoRA serves as the foundational building block of such parameter-efficient modular architectures, leveraging low-rank weight structures to reduce the number of trainable parameters. In this paper, we study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures. First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature and surprisingly outperforms both LoRA and full fine-tuning in our setting. Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks, thus scaling beyond what is usually studied in the literature. Our findings demonstrate that sparse adapters yield superior in-distribution performance post-merging compared to LoRA or full model merging. Achieving strong held-out performance remains a challenge for all methods considered.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。