TTRL模型在测试时会误杀正确答案，新方法有效防止这一问题。
Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting
- 通过翻转率监控识别正确答案消失窗口，动态调整学习策略。
- 在AIME 2025上相比原方法提升54%的解题准确率。
- 适合追求高可靠性的数学推理模型优化，尤其对小模型有效。
测试时强化学习（TTRL）在数学推理基准上利用多数投票作为伪标签信号，取得显著准确率提升。我们指出这些提升被系统性误解：多数反映的是已有可解问题的精度增强，而非真正学习；错误问题数量远超真正学会的问题，且一旦多数投票锁定错误答案，损伤不可逆。逐题追踪发现，低能力问题中的正确答案信号短暂活跃后即被永久压制，此现象称为“正确答案灭绝窗口”，翻转率（FR）是其关键指标。为此提出TTRL-Guard，包含三项机制：基于翻转率的奖励缩放（FRS）在FR下降时降低风险更新权重，少数保留采样（MPS）保留少数正确答案的梯度信号，风险条件稀疏更新（RCSU）暂停极化问题的更新。跨三个模型和四个基准的实验表明，TTRL-Guard在Qwen2.5-7B-Instruct和Qwen3-4B上达到最佳平均pass@1，相较TTRL在AIME 2025上相对提升54%。
原文摘要 · Abstract (English)
Test-time reinforcement learning (TTRL) reports substantial accuracy gains on mathematical reasoning benchmarks using majority vote as a pseudo-label signal. We argue these gains are systematically misinterpreted: most reflect sharpening of already-solvable problems rather than genuine learning, while problems corrupted from correct to incorrect outnumber truly learned ones, and this damage is irreversible once majority vote locks onto a wrong answer. Per-problem tracking reveals that correct-answer signals in low-ability problems are briefly active before being permanently suppressed, a phenomenon we term the \textit{Correct-Answer Extinction Window}, with Flip Rate (FR) as its leading indicator. We thus propose TTRL-Guard, a lightweight framework with three mechanisms targeting the extinction window: Flip-Rate-Aware Reward Scaling (FRS) down-weights at-risk updates as FR declines, Minority-Preserving Sampling (MPS) retains gradient signal from minority correct answers, and Risk-Conditioned Sparse Updatings (RCSU) suspends updates on polarized problems. Experiments across three models and four benchmarks show that TTRL-Guard achieves the best average pass@1 on Qwen2.5-7B-Instruct and Qwen3-4B, improves relatively over TTRL by +54\% on AIME 2025.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。