让LoRA自动分配每层训练参数量，更精准高效。
IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring
- 用积分梯度计算层内敏感度，比传统梯度更全面
- 引入动态校准机制，降低噪声对参数分配的影响
- 在多个模型和任务上表现优于现有方法，适合资源有限场景
随着大语言模型参数量达数十亿，全参数微调变得计算与内存不可行。参数高效微调（PEFT）通过仅更新少量任务相关参数来缓解此问题，其中低秩适应（LoRA）被广泛采用。但传统LoRA对各层使用统一秩，忽略了层间重要性差异。虽已有自适应秩方法（如AdaLoRA），但多依赖瞬时梯度，仅反映局部敏感性，忽略同层内的路径效应，导致评分不稳定且有偏。为此，我们提出IGU-LoRA：首先利用层内积分梯度（IG）敏感度并聚合为层级评分以分配秩；其次采用带偏差追踪的指数移动平均实现不确定性感知的校准机制，抑制噪声更新。理论上，在路径哈密顿-利普希茨条件下，我们证明了参数空间积分梯度的复合梯形法则近似误差上界，指导采样预算。在多种任务与架构下，IGU-LoRA在相同参数预算下持续超越强基线，提升下游准确率与鲁棒性。消融实验验证了路径内敏感度估计与不确定性感知选择的有效性。代码已公开于 https://github.com/withyou12/igulora.git。
原文摘要 · Abstract (English)
As large language models (LLMs) scale to billions of parameters, full-parameter fine-tuning becomes compute- and memory-prohibitive. Parameter-efficient fine-tuning (PEFT) mitigates this issue by updating only a small set of task-specific parameters while keeping the base model frozen. Among PEFT approaches, low-rank adaptation (LoRA) is widely adopted; however, it enforces a uniform rank across layers despite substantial variation in layer importance, motivating {layerwise} rank allocation. Recent adaptive-rank variants (e.g., AdaLoRA) allocate ranks based on importance scores, yet typically rely on instantaneous gradients that capture only local sensitivity, overlooking non-local, pathwise effects within the same layer, which yields unstable and biased scores. To address this limitation, we introduce IGU-LoRA, an adaptive-rank LoRA that (i) computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation, and (ii) applies an uncertainty-aware scheme using exponential moving averages with deviation tracking to suppress noisy updates and calibrate rank selection. Theoretically, we prove an upper bound on the composite trapezoidal rule approximation error for parameter-space IG under a pathwise Hessian-Lipschitz condition, which informs the quadrature budget. Across diverse tasks and architectures, IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness. Ablations confirm the contributions of pathwise within-layer sensitivity estimates and uncertainty-aware selection to effective rank allocation. Our code is publicly available at https://github.com/withyou12/igulora.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。