无需训练即可高效分配LoRA秩,提升模型适应能力
RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation

- 通过前向传播探测表示敏感度,动态分配适配秩
- 在三个基准上平均性能最优,配置速度提升1.48倍以上
- 兼容多种高效微调方法,适合追求效率的部署场景
参数高效微调使大语言模型以更低计算与存储成本适配下游任务,其中低秩适配(LoRA)应用最广。但传统LoRA对所有模块统一分配秩,现有自适应方法或需额外优化开销,或依赖静态权重和局部梯度,无法捕捉任务相关的表征变化。本文提出RSRA,一种无需训练的秩分配器,通过小规模校准集的前向传播探测表示敏感度,实现任务感知的秩配置。具体地,RSRA使用谱有效秩跨层分配容量,采用弗雷歇距离衡量标准化虚拟低秩更新下的模块级隐藏状态偏移,并通过分层归一化融合两者信号。在Qwen3-4B与Mistral-7B上针对常识推理与自然语言理解任务测试,RSRA在所有三个模型-基准组合中达到最高平均性能,且分配时间比最快对比方法快1.48至1.93倍。集成至DoRA、LoRA-FA和PiSSA后,改善了18组中的15组表现,显著提升三类参数高效微调方法的平均性能。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning enables large language models to adapt to downstream tasks with substantially lower computational and storage cost, and Low-Rank Adaptation (LoRA) is among its most widely used techniques. However, vanilla LoRA assigns a uniform rank to all adapted modules, while existing adaptive methods either incur additional optimization overhead or rely on static weights and local gradients that do not capture task-conditioned representation changes. We propose RSRA, a training-free rank allocator that estimates where adaptation capacity is most needed through forward-only representation sensitivity probing on a small calibration set. Specifically, RSRA uses Spectral Effective Rank to allocate capacity across layers, measures module-wise hidden-state displacement under standardized virtual low-rank updates with the Frechet Distance, and combines both signals through hierarchical normalization to produce a task-aware rank configuration before fine-tuning. Across commonsense reasoning and natural language understanding benchmarks with Qwen3-4B and Mistral-7B, RSRA achieves the highest average performance in all three reported model-benchmark settings and a 1.48x-1.93x speedup in allocation time over the fastest competing pre-allocation method. When integrated with DoRA, LoRA-FA, and PiSSA, RSRA improves 15 of the 18 evaluated combinations and increases the average performance of all three PEFT methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。