CopRA通过渐进式训练提升LoRA的泛化能力，支持高效模型合并与剪枝。
CopRA: A Progressive LoRA Training Strategy
- 采用分层随机丢弃的渐进训练策略，优化各层参数贡献度。
- 训练后参数具线性模式连通性，支持高效模型合并。
- 适用于联邦学习、多任务学习及模型剪枝场景。
低秩适配（LoRA）是一种高效的微调方法，但标准训练常快速收敛至初始化附近的局部最优，可能不利于分布外数据或模型合并、剪枝等任务。本文提出一种新的渐进式训练策略——合作型LoRA（CopRA），结合随机层丢弃并优化每层参数的Shapley值，将各层视为合作博弈中的参与者。实验表明，CopRA训练出的参数具有线性模式连通性，支持高效模型合并，为基于LoRA的联邦学习与多任务学习提供可能；同时在剪枝任务中表现更优。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is a parameter-efficient technique for rapidly fine-tuning foundation models. In standard LoRA training dynamics, models tend to quickly converge to a local optimum near the initialization. However, this local optimum may not be ideal for out-of-distribution data or tasks such as merging and pruning. In this work, we propose a novel progressive training strategy for LoRA with random layer dropping. This strategy also optimizes the Shapley value of LoRA parameters in each layer, treating each layer as a player in a cooperative game. We refer to this method as Cooperative LoRA (CopRA). Our experimental results demonstrate that parameters trained with CopRA exhibit linear mode connectivity, which enables efficient model merging. This also paves the way for federated learning and multi-task learning via LoRA merging. Additionally, by optimizing the Shapley value, CopRA shows superior performance in pruning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。