用重述一致性筛选答案，让模型推理更稳定可靠
Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning
- 单模型分角色扮演解题者和重述者，通过重述一致性判断答案真伪
- 用调和平均代替多数投票，使正确答案在不同表述下保持稳定
- 无需人工标注或额外模型，跨28个测试场景均表现最优
测试时强化学习（TTRL）提供了一种无需标签的推理适应范式，仅依赖推理阶段的合成信号。然而其效果取决于学习信号的可靠性。传统方法如多数投票常陷入虚假但流行的答案。本文提出Self-Harmony框架，核心思想是：正确答案在原问题及其重述下应保持一致。该框架使用单一模型承担两个角色：解题者生成答案，重述者重构输入。进一步提出伪标签方法：不采用多数投票，而是对原始与重述视图的答案频率使用调和平均进行聚合。这一过程自然筛选出在重述下稳定的解法，避免选择依赖特定视角的虚假答案。关键优势在于无需人类标注或辅助模型。在多个推理基准上，Self-Harmony在30个设置中取得28个第一，实现当前最优性能。此外，所有实验中零训练失败，展现出前所未有的鲁棒性与可靠性。
原文摘要 · Abstract (English)
Test-time reinforcement learning (TTRL) offers a label-free paradigm for adapting models using only synthetic signals at inference, but its success hinges on constructing reliable learning signals. Standard approaches such as majority voting often collapse to spurious yet popular answers. We introduce Self-Harmony, a framework built on a simple intuition: the correct answer should remain stable across both an original question and its paraphrase. Self-Harmony operationalizes this by employing a single model in two complementary roles: a Solver to produce answers and a Reframer to rephrase the input. Based on this, we further propose a pseudo-label method: instead of majority voting, it aggregates answer frequencies across these original and reframed views using the harmonic mean. This is a process that naturally selects for solutions stable under reframing, thereby avoiding the common trap of favoring view-dependent, spurious answers. Crucially, this requires no human supervision or auxiliary models. Across diverse reasoning benchmarks, Self-Harmony achieves state-of-the-art results at the label-free test-time setting, ranking first in 28 of 30 settings across multiple methods. Beyond accuracy, it demonstrates unprecedented robustness, with zero training failures in all experiments, underscoring its stability and reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。