让大模型在测试时更聪明：通过筛选共识与自适应纠错提升推理可靠性
What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
- 只信任强共识，拒绝模糊多数；用熵值判断错误路径并主动剔除
- 在多个推理任务上超越基线，即使在有限尝试次数下仍稳定提升
- 首次引入负样本监督，适合需要高可靠性的复杂推理场景
测试时强化学习（TTRL）使大语言模型通过多数投票生成伪奖励，从而增强对未标注测试流的推理能力。然而，现有方法仅依赖正向伪标签，在答案分布分散时易因弱共识错误引导模型。本文提出SCRL框架，包含选择性正伪标签机制，通过严格共识标准过滤不可靠多数；同时引入基于熵的负伪标签机制，首次在TTRL中实现负向监督，依据生成不确定性可靠剔除错误路径。大量实验表明，SCRL在多个推理基准上显著优于基线，在受限回放预算下仍保持良好泛化与训练稳定性。
原文摘要 · Abstract (English)
Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。