arXiv:2605.08423cs.LGcs.CL2026-05被引 1

让低秩微调动态适应输入,提升模型表现与稳定性。

Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms

论文配图:Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms
图 1 · 摘自论文原文
  • 用可查询的记忆库动态选择低秩更新组件,替代固定结构。
  • 在非线性回归和大模型微调中,测试性能与训练更稳定。
  • 适合追求高效微调且需上下文感知更新的开发者使用。

我们提出一种数据自适应的参数高效微调方法。标准低秩适配(LoRA)将每层更新限制为固定低秩形式,但在输入变化或网络深度计算演化时可能过于僵化。本文方法将局部适配器替换为共享的可查询更新原子记忆体:每个层块基于当前低秩状态和前序块的汇总信息生成查询,通过注意力机制检索相关内容依赖的共享更新成分,并在低秩瓶颈中应用路由算子。该设计保持了低秩适配的效率与可扩展性,同时支持跨输入变化的动态更新,并实现层间结构复用。进一步引入指令正则化,通过语言先验增强路由逻辑,引导选择语义相关的低秩变换方向,避免生成无约束参数。在噪声非线性回归任务与大语言模型微调实验中,该方法相比标准低秩适配,在使用相近可训练参数量的情况下,提升了最终测试性能与训练稳定性。

原文摘要 · Abstract (English)

We present a data-adaptive method for parameter-efficient fine-tuning of large neural networks. Standard low-rank adaptation methods improve efficiency by restricting each layer update to a fixed low-rank form, but this static parameterization can be too rigid when the appropriate correction depends on the input and on the evolving depth-wise computation of the network. Our approach replaces a purely layer-local adapter with a shared queryable memory of low-rank update atoms. For each block of layers, the model forms a query from the current low-rank state and a running summary of previous blocks, uses this query to retrieve a content-dependent combination of shared update components via attention, and applies the resulting routed operator within the low-rank bottleneck. In this way, the method retains the efficiency and scalability of low-rank adaptation while allowing the effective update to vary across inputs and to share reusable structure across layers. The resulting architecture provides a principled middle ground between static LoRA-style updates and fully generated parameter updates: it remains compact and parameter-efficient while supporting dynamic, context-sensitive adaptation. Further, we incorporate instruction-regularization by augmenting routing logits with a language-induced prior over update atoms, thereby biasing the selection of low-rank transformations toward semantically relevant directions without generating unconstrained parameter updates. Experiments on noisy non-linear regression tasks and LLM fine-tuning suggest that this queryable update-memory formulation can improve final test performance and training stability compared to standard low-rank adaptation, while using a comparable number of trainable parameters.

参数高效低秩微调动态路由大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。