用外部工具验证提升测试时强化学习的可靠性，防止错误答案因高频出现被误认正确。
Tool Verification for Test-Time Reinforcement Learning
- 引入外部工具证据验证采样轨迹，通过可信度加权改进多数投票机制。
- 在MATH-500、AMC和AIME 2024上显著优于传统TTRL，尤其在难题上提升明显。
- 适合需要高可靠在线自适应的场景，如数学推理与代码生成任务。
测试时强化学习（TTRL）通过在未标注测试输入上利用大推理模型进行自我改进，基于采样轨迹的多数投票生成自一致性奖励。然而，多数投票可能将流行性误判为正确性：一个虚假但高频的未验证共识会成为有偏奖励信号，导致TTRL强化错误答案并陷入错误模式。本文提出T³RL（Tool-Verification for Test-Time Reinforcement Learning），一种验证感知的测试时强化学习框架。T³RL将伪标签构建锚定于外部工具证据（如代码执行）。具体而言，验证器利用外部工具证据对采样轨迹进行可信度评估，并在验证感知投票中提高已验证轨迹的权重，从而生成更可靠的伪标签用于训练。在MATH-500、AMC和AIME 2024等多个数学难度数据集及多种骨干模型上，T³RL均显著优于传统TTRL，尤其在更难问题上表现更优。T³RL定位为一种经过验证的在线数据合成器，凸显了工具验证在可靠在线适应中的关键作用。该方法可扩展至其他可验证领域，且具备训练与推理效率优势。
原文摘要 · Abstract (English)
Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for Recursive Self-Improving AI (RSI) by adapting Large Reasoning Models (LRMs) on unlabeled test inputs, using self-consensus rewards derived from majority voting over sampled rollouts. However, majority voting can mistake popularity for correctness: a spurious yet high-frequency unverified consensus may become a biased reward signal, causing test-time RL to reinforce frequent but wrong answers and collapse into an incorrect mode. We address this false-popular failure mode with T$^3$RL (Tool-Verification for Test-Time Reinforcement Learning), a verification-aware test-time RL framework. T$^3$RL grounds pseudo-label construction in external tool evidence. Concretely, a verifier utilizes external tool evidence (e.g., from code execution) to upweight verified rollouts during a verification-aware voting, producing more reliable pseudo-labels for training. Across various math difficulties (MATH-500, AMC, and AIME 2024) and diverse backbone families, T$^3$RL significantly improves over TTRL, with better performance on harder problems. T$^3$RL is positioned as a verified online data synthesizer, highlighting the role of tool verification in reliable online adaptation. T$^3$RL is extensible to verifiable domains and provides training and inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。