通过固定奇异基避免主成分干扰，实现高效持续学习
TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
- 用预训练权重的奇异基作参考框架，仅更新奇异值矩阵
- 软谱惩罚抑制主方向更新，降低任务间干扰
- 适合资源受限场景下的持续学习，尤其关注参数效率
基于谱分解的参数高效微调方法推动了持续学习的发展。本文提出TailLoR，利用预训练权重的奇异基U和V作为固定参考系，对奇异值矩阵施加低秩更新。通过引入软谱惩罚，抑制与主要奇异方向对齐的更新，从而减少干扰，并将精细调整引导至高灵活性的长尾谱坐标中。该方法在保持模型性能的同时，显著提升了参数效率与任务连续性。
原文摘要 · Abstract (English)
Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。