让大模型推理更精准,通过动态调整适配器和选择相关上下文。
Selective State-Space Adaptation and Retrieval for Language Model Reasoning

- 在词元级别用递归状态动态调节适配器,实现输入相关更新。
- 上下文级别检索选相关段落,三百万参数的模型超越八亿参数密集检索器。
- 适用于多跳推理任务,可显著提升冻结模型的推理准确率。
低秩适配(LoRA)引入静态学习更新,对所有输入一致应用,仅提供任务级适配而未显式建模词元或实例级状态变化。本文提出一类双粒度选择性状态空间控制适配器:在词元级,MaLoRA(Mamba调制低秩适配)将适配器缩放因子设为依赖输入的动态函数,并保留跨词元的递归状态,优于传统无状态调制器;在上下文级,MaRA(Mamba检索适配器)追踪跨段推理状态并选择最相关的段落。约三百万参数的状态空间控制检索,在支持段落召回上超过八亿参数的密集检索器。基础模型在无适配下表现差(F1 14~25),MaRA恢复其表示中的证据相关性。在三个冻结主干模型和两个多跳推理基准上,该方法在3×2网格中每项均提升推理准确率,平均比LoRA基线提升6.4 F1(相对+10.0%)。
原文摘要 · Abstract (English)
Low-rank adaptation introduces a static learned update applied identically to every input. The update provides task-level adaptation but does not explicitly represent token-level or instance-level state variation. A family of adapters is proposed that introduces selective state-space control at two complementary granularities. At the token level, MaLoRA (Mamba-modulated low-rank adaptation) makes the adapter's scaling factor a dynamic input-dependent function with recurrent state across tokens, in contrast to the stateless modulators of prior work. The token-level adapter improves over low-rank adaptation. On the other hand, it differentiates tokens by structural role but not by contextual relevance, which motivates placing evidence selection at the context level. At the context level, MaRA (Mamba Retrieval Adapter) tracks cross-segment reasoning state and selects the segments most relevant to the query. State-space controlled retrieval of approximately three million parameters exceeds an eight-billion-parameter dense retriever on supporting-paragraph recall. Although base models perform poorly on the task without adaptation (14 to 25 F1), MaRA recovers the evidence relevance latent in their representations. Across three frozen backbones and two multi-hop reasoning benchmarks, the end-to-end family improves reasoning accuracy on every cell of the 3-by-2 grid, by +6.4 F1 (+10.0% relative) on average over the LoRA baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。