通过渐进激活提升LoRA微调效果，改善模型泛化与可合并性。
Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
- 渐进式提高适配器激活概率，避免陷入局部最优。
- 在多任务场景下提升合并准确率2.3个百分点，修剪后性能更稳定。
- 适用于各类LoRA变体，适合追求高效微调的开发者。
低秩适应（LoRA）是主流的参数高效微调方法，但常因适配器陷入初始化附近的次优极小值而限制模型泛化能力，影响适配器合并与剪枝等下游操作。本文提出CoTo，一种渐进训练策略，逐步增加适配器的激活概率。通过随机禁用适配器，该方法促进更均衡的优化并扩大损失曲面的探索范围。理论分析表明，CoTo能增强层间丢弃稳定性与线性模式连通性，并采用合作博弈方法量化各适配器的边际贡献。大量实验表明，CoTo在单任务上持续提升性能，在多任务合并中准确率提升2.3%，增强剪枝鲁棒性，降低训练开销，且兼容多种LoRA变体。代码已公开于https://github.com/zwebzone/coto。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) has emerged as a leading parameter-efficient fine-tuning technique for adapting large foundation models, yet it often locks adapters into suboptimal minima near their initialization. This hampers model generalization and limits downstream operators such as adapter merging and pruning. Here, we propose CoTo, a progressive training strategy that gradually increases adapters' activation probability over the course of fine-tuning. By stochastically deactivating adapters, CoTo encourages more balanced optimization and broader exploration of the loss landscape. We provide a theoretical analysis showing that CoTo promotes layer-wise dropout stability and linear mode connectivity, and we adopt a cooperative-game approach to quantify each adapter's marginal contribution. Extensive experiments demonstrate that CoTo consistently boosts single-task performance, enhances multi-task merging accuracy, improves pruning robustness, and reduces training overhead, all while remaining compatible with diverse LoRA variants. Code is available at https://github.com/zwebzone/coto.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。