为LoRA设计自适应学习率，让每组低秩更新更精准高效。
One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
- 为每个低秩组件动态分配学习率，基于训练信号在线调整。
- 在多个任务上超越标准LoRA，且在不同学习率下表现稳定。
- 无需额外参数，可无缝适配其他LoRA变体，适合高效微调场景。
低秩适配（LoRA）已成为大语言模型参数高效微调的标准方法。多数LoRA变体采用统一学习率，对所有低秩组件使用单一全局学习率。我们发现这一惯例忽略了模块内部的显著异质性：同一LoRA适配器中的低秩组件更新速率差异巨大，低速模块收敛于集中奇异谱，导致名义秩预算未被充分利用。为此，我们提出一种自适应各向异性学习率模型，为每个低秩组件分配独立的有效学习率，该值基于训练时信号在线计算，并按模块均值归一化以维持全局学习率预算。AnLR-LoRA在此模型基础上，利用AdamW优化过程中可得的函数空间速度与Adam SNR两个信号，实现轻量级方案，无额外可训练参数。在常识推理、自然语言生成及视觉指令微调等基准测试中，AnLR-LoRA持续优于标准LoRA，同时促进秩容量更广泛使用，性能增益在宽范围全局学习率下保持稳健，并可迁移至其他LoRA变体。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。