ReCoLoRA让大模型持续学新任务时不遗忘旧任务，还能少训练参数。
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
- 基于谱分析动态选择每层有效秩，递归融合前序任务更新
- 六任务连续微调中三项指标优于现有方法，训练参数更少
- 适合需要长期学习新任务的大模型应用
参数高效微调能低成本适应单个任务，但在连续任务序列中，类似LoRA的方法会不断叠加低秩更新，导致新任务覆盖旧任务。本文提出ReCoLoRA（递归整合低秩适配器），一种谱感知的持续微调框架：从预训练权重的随机奇异值分解初始化适配器，通过肘部准则选择每层有效秩，并优先在主子空间中更新，再开放残差容量。每次新任务前，ReCoLoRA对当前有效权重进行重新分解，而非原始权重，生成冻结残差、缓慢更新的主成分和全新适配器（递归整合），使每个任务均基于已吸收前序任务的模型开始。在四个7-8B规模骨干网络上的六任务连续GLUE序列测试中，ReCoLoRA在三个骨干网络上达到最优平均分数，优于秩扫描的LoRA、PiSSA、AdaLoRA和DoRA基线，且训练参数更少；一个基于路由的任务库变体作为全任务隔离下的上限基准。代码见：https://github.com/bhqy666/ReCoLoRA。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning adapts a large language model to one task cheaply, but across a task sequence LoRA-style methods keep stacking low-rank updates on the same frozen weight, so each new task tends to overwrite the previous ones. We present ReCoLoRA (Recursive Consolidation of Low-Rank Adapters), a spectrum-aware framework for continual fine-tuning: adapters are initialized from a randomized SVD of the pretrained weight, per-layer effective ranks are selected by an elbow criterion, and the principal subspace is adapted before residual capacity is opened. Before each new task, ReCoLoRA re-decomposes the current effective weight, rather than the original one, into a frozen residual, a slowly updated principal component, and a fresh adapter (recursive consolidation), so every task starts from the model that has already absorbed its predecessors. On a six-task continual GLUE sequence over four 7-8B backbones, ReCoLoRA attains the best final average score on three of the four backbones against rank-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines while training fewer parameters; an oracle-routed task-bank variant serves as an upper bound under full task isolation. Code: https://github.com/bhqy666/ReCoLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。