通过聚焦参数更新的主方向，让模型更省力地记住新任务。
Energy-Structured Low-Rank Adaptation for Continual Learning
- 利用参数更新的低秩特性，将知识集中到主要方向上。
- 在多个基准测试中超越现有方法，实现最佳持续学习性能。
- 适合需要长期学习新知识且资源受限的场景。
尽管正交子空间方法试图缓解持续学习中的任务干扰，但常因基底间的能量扩散导致知识分散，耗尽未来任务的容量。我们观察到参数更新引发的输出特征漂移本质上是低秩的，并从理论上证明：沿该漂移主方向保留参数可最小化输出重构误差。受此启发，提出能量集中与有序的低秩适配方法（E²-LoRA）。通过显式对齐并集中知识至前导秩，E²-LoRA释放出更多容量以应对后续任务。此外，设计动态秩分配策略，在保持能量与模型灵活性之间取得平衡。大量实验表明，E²-LoRA在多个基准上达到当前最优性能。代码已开源：https://github.com/kiddo127/E2-LoRA。
原文摘要 · Abstract (English)
While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose \textbf{E}nergy-Concentrated and \textbf{E}nergy-Ordered \textbf{Lo}w-\textbf{R}ank \textbf{A}daptation (E$^2$-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E$^2$-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E$^2$-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。