不重新训练，通过重加权奇异值提升LoRA性能。
Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting
- 用SVD分解LoRA更新，按梯度敏感度重加权奇异值。
- 在多个任务上提升4.4分（CommonsenseQA）和2.4%（HumanEval）。
- 仅修改约1000个标量，适合快速优化已训练LoRA模型。
低秩适配（LoRA）通过限制任务更新至低秩参数子空间来提升下游性能，但训练后适配器内部的容量分配机制尚不清晰。我们通过跨多任务与主干网络的几何与实证研究发现，训练好的LoRA更新常呈现谱效率低下：任务影响集中于少数奇异方向，其余成分中性甚至有害，因而需要对学习到的子空间进行事后优化。本文提出Spectral Surgery，一种无需训练的精调方法：将LoRA更新进行SVD分解，利用小校准集上的梯度估计各分量敏感度，并在幅值约束下重加权奇异值，同时固定已学方向。在Llama-3.1-8B与Qwen3-8B上四个基准测试中，该方法仅调整约1000个标量系数，即实现一致提升——CommonsenseQA最高+4.4分，HumanEval pass@1最高+2.4点。结果表明，基于SVD结构的低成本参数编辑可作为纯事后方式有效改进已训练的LoRA适配器。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace, yet how this limited capacity is allocated within a trained adapter remains unclear. Through a geometric and empirical study across multiple tasks and backbones, we find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions, while many remaining components are neutral or detrimental, motivating post-hoc refinement within the learned subspace. We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed. Across Llama-3.1-8B and Qwen3-8B on four benchmarks, Spectral Surgery yields consistent gains (up to +4.4 points on CommonsenseQA and +2.4 pass@1 on HumanEval) by adjusting only $\approx 1{,}000$ scalar coefficients. These results demonstrate that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。