GRIT让小参数微调更智能，自动适应模型几何结构。
GRIT -- Geometry-Aware PEFT with K-FACPreconditioning, Fisher-Guided Reprojection, andDynamic Rank Adaptation
- 用K-FAC预处理梯度，结合费舍尔特征方向动态重投影。
- 根据损失曲率自适应调整低秩维度，减少无效参数更新。
- 适合追求高效微调且避免性能退化的研究人员。
参数高效微调（PEFT）是适配大语言模型的主流方法，但广泛使用的LoRA和QLoRA largely 忽略了局部损失曲率，优化时在固定随机方向的低秩子空间中进行一阶下降，可能导致有效更新预算膨胀，并放大弱约束方向上的漂移。我们提出GRIT，一种动态、曲率感知的LoRA方法，在保持原有参数化形式的基础上：(1) 使用K-FAC作为自然梯度代理，对秩空间中的梯度进行预处理；(2) 定期将低秩基向量投影到主导费舍尔特征方向，抑制漂移；(3) 根据谱分布自适应调整有效秩，使容量集中于信号显著区域。在LLaMA系列模型上，针对指令遵循、理解与推理任务的多项基准测试显示，GRIT在平均减少46%可训练参数（任务间25%-80%）的同时，性能不低于或优于LoRA与QLoRA，且在不同提示风格与数据组合下无明显质量损失。为建模遗忘现象，我们拟合了一个曲率调制的幂律模型。实证表明，相比强基线方法（如Orthogonal-LoRA、IA3、DoRA、Eff-FT、Shampoo），GRIT具有更低的漂移水平和更优的更新-保留权衡表现。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) is the default way to adapt LLMs, but widely used LoRA and QLoRA are largely geometry-agnostic: they optimize in fixed, randomly oriented low-rank subspaces with first-order descent, mostly ignoring local loss curvature. This can inflate the effective update budget and amplify drift along weakly constrained directions. We introduce GRIT, a dynamic, curvature-aware LoRA procedure that preserves the LoRA parameterization but: (1) preconditions gradients in rank space using K-FAC as a natural-gradient proxy; (2) periodically reprojects the low-rank basis onto dominant Fisher eigendirections to suppress drift; and (3) adapts the effective rank from the spectrum so capacity concentrates where signal resides. Across instruction-following, comprehension, and reasoning benchmarks on LLaMA backbones, GRIT matches or surpasses LoRA and QLoRA while reducing trainable parameters by 46% on average (25--80% across tasks), without practical quality loss across prompt styles and data mixes. To model forgetting, we fit a curvature-modulated power law. Empirically, GRIT yields lower drift and a better updates-vs-retention frontier than strong PEFT-optimizer baselines (Orthogonal-LoRA, IA3, DoRA, Eff-FT, Shampoo).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。