arXiv:2511.17938cs.CLcs.LG2025-11被引 11

让大模型推理更稳:只优化关键决策点,避免回答变短

SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization

  • 只更新推理路径中的关键决策词元,不盲目调整所有词元
  • 在8个任务上提升通过率(Pass@1),且回答长度不缩水
  • 无需标签或奖励模型,适合想稳定改进推理的开发者

大型语言模型和多模态语言模型在链式思维推理中表现优异,但在测试时面临分布偏移且缺乏可验证监督。现有测试时强化学习方法通过采样轨迹的自一致性投票生成无标签伪奖励,但常出现崩溃:多数票奖励主导,响应缩短,通过率下降。我们发现根源在于对整个序列进行均匀更新,多数低熵词元为跟随者,仅少数高熵词元决定推理分支。为此,我们提出 extit{SPINE},一种词元选择性测试时强化学习框架,(i)基于分布感知的分支点选择机制,仅更新关键决策词元;(ii)在这些词元上施加鲁棒的熵带正则化,防止过早崩溃并抑制噪声漂移。该方法可集成至GRPO类目标函数(可选KL锚点),无需标签或奖励模型。在涵盖多模态视觉问答与纯文本推理的八个基准上, extit{SPINE}持续优于传统TTRL,避免响应长度收缩,且在LLM与MLLM骨干网络上均实现更稳定的训练动态。结果表明,将更新对齐于链式思维分支点是一种简单、无标签的稳定测试时适应机制。

原文摘要 · Abstract (English)

Large language models (LLMs) and multimodal LLMs (MLL-Ms) excel at chain-of-thought reasoning but face distribution shift at test-time and a lack of verifiable supervision. Recent test-time reinforcement learning (TTRL) methods derive label-free pseudo-rewards from self-consistency voting over sampled trajectories, yet they often collapse: the majority-vote reward prevails, responses shorten, and Pass@1 declines. We trace this to uniform sequence updates in which most tokens are low-entropy followers, while a small high-entropy subset determines the reasoning branches. Thus we propose \method, a token-selective test-time reinforcement learning framework that (i) performs distribution-aware forking-token selection to update only decision-critical branch points, and (ii) applies a robust entropy-band regularizer at those tokens to prevent premature collapse and suppress noisy drift. \method plugs into GRPO-style objectives (optionally with a KL anchor) and requires neither labels nor reward models. Across eight benchmarks spanning multimodal VQA, text-only reasoning, \method consistently improves Pass@1 over TTRL while avoiding response-length collapse and yielding more stable training dynamics on both LLM and MLLM backbones. These results indicate that aligning updates with chain-of-thought branch points is a simple and label-free mechanism for stable and effective test-time adaptation in reasoning models. Code will be released.

强化学习推理优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。