通过局部低秩结构提升微调效率，性能优于传统方法。
Localized LoRA: A Structured Low-Rank Approximation for Efficient Fine-Tuning
- 将权重更新分解为结构化块的低秩矩阵组合
- 相同参数量下逼近误差更低，提升模型表达能力
- 适合需要高效微调的场景，如资源受限部署
参数高效微调（PEFT）方法如LoRA通过引入低秩更新来替代全模型微调，实现轻量化。然而现有方法多依赖全局低秩结构，忽略参数空间中的局部模式分布。本文提出局部LoRA，将权重更新建模为作用于权重矩阵结构化块的低秩矩阵组合，可在不增加可训练参数总量的前提下实现密集、局部化的更新。我们形式化对比了全局、对角局部与完全局部低秩近似，证明所提方法在相同参数预算下始终具有更低的近似误差。合成数据与实际任务实验表明，局部LoRA在表达性与适应性上均优于现有方法，实现更高效的微调并提升性能。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, offer compact and effective alternatives to full model fine-tuning by introducing low-rank updates to pre-trained weights. However, most existing approaches rely on global low rank structures, which can overlook spatial patterns spread across the parameter space. In this work, we propose Localized LoRA, a generalized framework that models weight updates as a composition of low-rank matrices applied to structured blocks of the weight matrix. This formulation enables dense, localized updates throughout the parameter space without increasing the total number of trainable parameters. We provide a formal comparison between global, diagonal-local, and fully localized low-rank approximations, and show that our method consistently achieves lower approximation error under matched parameter budgets. Experiments on both synthetic and practical settings demonstrate that Localized LoRA offers a more expressive and adaptable alternative to existing methods, enabling efficient fine-tuning with improved performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。