无需标签，在测试时优化大模型推理能力，效果媲美有监督方法。
TTPO: Test-Time Policy Optimization
- 利用伪标签与不对称目标，区分正确和错误推理路径。
- 在五个竞赛级基准上达到与有标签方法相当的性能，提升模型准确率7.2个百分点。
- 适合追求零标注测试时优化的AI研究者与开发者使用。
近期主流的后训练方法如强化学习（RL）和在线策略自蒸馏（OPSD）推动了大语言模型在数学推理上的快速进步，但其依赖真实标签，无法进行测试时训练（TTT）。用多数投票伪标签替代真实标签虽自然，却脆弱：一次错误投票会污染教师模型并误导所有生成项。我们观察到此失败模式具有不对称性：与伪标签不一致的推理路径通常本身错误，无论投票是否正确。基于此，提出测试时策略优化（TTPO），采用不对称目标，通过OPSD蒸馏一致推理路径，同时以分组强化学习惩罚不一致路径。分词级选择进一步优化：蒸馏降低已收敛位置权重，强化学习仅惩罚高置信度错误。两项更新在伪标签频繁出错时仍保持稳健，随着模型改进，多数投票路由实现更紧的自监督。无需任何标签，TTPO在五项竞赛级基准上匹配有标签监督的OPSD表现，使Qwen3-1.7B模型在测试时推理准确率从38.0%提升至45.2%，无思考场景下提升25.2%至36.4%，展现出强跨任务泛化能力。
原文摘要 · Abstract (English)
Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。