arXiv:2602.12846cs.LGcs.AI2026-02被引 1

提出ARTS方法,让大模型推理更全面,不丢掉罕见但正确的解题路径。

Amortized Reasoning Tree Search: Decoupling Proposal and Decision in Large Language Models

  • 将生成与验证分离,用流匹配优化稀疏搜索空间中的概率传递
  • 在MATH-500上达74.6%准确率,接近全微调效果
  • 显著提升长尾难题表现,避免传统强化学习导致的路径消失

强化学习结合可验证奖励(RLVR)已成为赋予大语言模型严谨推理能力的主要范式。然而,我们发现该对齐过程存在关键缺陷:系统性抑制了有效但罕见(基模型分布下低概率)的推理路径。理论上,这被描述为“归一化挤压”现象,即模式导向的策略梯度与有限采样共同作用,形成高通量似然滤波器,使稀有正确推理路径的概率趋于统计灭绝。为应对这一崩溃,同时保留基模型的潜在多样性,我们提出折衷推理树搜索(ARTS)。不同于通过参数更新内化推理的标准方法,ARTS优先考虑思辨过程,将生成与验证解耦。引入流匹配目标,重新利用验证器估计概率流守恒,从而在传统判别目标失效的稀疏、高熵搜索空间中实现稳健导航。在MATH-500基准上的大量实验表明,ARTS达到74.6%(BoN@16),与完全微调策略(74.7%)性能相当,且未修改生成主干。关键的是,在长尾子集上,耦合强化学习优化退化至0% pass@k,而ARTS独特地恢复了显著性能,表明将验证与生成解耦是解决复杂推理任务更稳健的路径。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has established itself as the dominant paradigm for instilling rigorous reasoning capabilities in Large Language Models. While effective at amplifying dominant behaviors, we identify a critical pathology in this alignment process: the systematic suppression of valid but rare (low-likelihood under the base model distribution) reasoning paths. We theoretically characterize this phenomenon as a "Normalization Squeeze," where the interplay between mode-seeking policy gradients and finite sampling acts as a high-pass likelihood filter, driving the probability of rare correct traces to statistical extinction. To counteract this collapse without discarding the base model's latent diversity, we propose Amortized Reasoning Tree Search (ARTS). Unlike standard approaches that force internalization via parameter updates, ARTS prioritizes deliberation by decoupling generation from verification. We introduce a Flow Matching objective that repurposes the verifier to estimate the conservation of probability flow, enabling robust navigation through sparse, high-entropy search spaces where traditional discriminative objectives fail. Extensive experiments on the MATH-500 benchmark demonstrate that ARTS achieves a performance of 74.6% (BoN@16), effectively matching fully fine-tuned policies (74.7%) without modifying the generative backbone. Crucially, on the long-tail subset where coupled RL optimization collapses to 0% pass@k, ARTS uniquely recovers significant performance, suggesting that disentangling verification from generation offers a more robust pathway for solving complex reasoning tasks.

推理增强强化学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。