用谱编码提升LoRA效率，减少冗余参数不降性能
Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
- 用谱基重构LoRA，从稀疏谱子空间重参数化
- 参数更少但表现更好，多任务上超越强基线
- 简单易插即用，适合想优化LoRA的开发者
低秩适配（LoRA）已成为微调大模型的主流方法。尽管成效显著，其庞大的参数冗余限制了模型容量与效率，被视为瓶颈。本文系统研究了微调中冗余的影响，发现降低密度冗余不会削弱表达能力。基于此，提出谱编码低秩适配（SeLoRA），利用谱基的强表达性，将LoRA重参数化至稀疏谱子空间。设计简洁，可无缝集成各类LoRA变体以提升性能，作为可扩展的即插即用框架。大量实验表明，SeLoRA以更少参数实现更高效率，在常识推理、数学推理和代码生成等下游任务上均优于强基线。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has emerged as a prominent technique for fine-tuning large foundation models. Despite its successes, the substantial parameter redundancy, which limits the capacity and efficiency of LoRA, has been recognized as a bottleneck. In this work, we systematically investigate the impact of redundancy in fine-tuning LoRA and reveal that reducing density redundancy does not degrade expressiveness. Based on this insight, we introduce \underline{S}pectral-\underline{e}ncoding \underline{L}ow-\underline{R}ank \underline{A}daptation (SeLoRA), which harnesses the robust expressiveness of spectral bases to re-parameterize LoRA from a sparse spectral subspace. Designed with simplicity, SeLoRA enables seamless integration with various LoRA variants for performance boosting, serving as a scalable plug-and-play framework. Extensive experiments substantiate that SeLoRA achieves greater efficiency with fewer parameters, delivering superior performance enhancements over strong baselines on various downstream tasks, including commonsense reasoning, math reasoning, and code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。