用测试时强化学习让大模型真实展现能力，避免误判熟悉度带来的假象。
Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks
- 测试时分两阶段强化学习，自动对齐任务格式与基准分布。
- 无需特定训练数据，效果接近监督微调，但更稳定可靠。
- 揭示许多推理提升实为熟悉度伪装，适合评估模型真实能力。
直接在基准上评估大语言模型可能产生误导，因为表现优异可能源于任务熟悉而非真实能力。传统的‘先训练后测试’方法通过任务相关微调控制熟悉度，但高质量训练数据难获取，结果受数据影响大。本文提出一种两阶段测试时强化学习对齐方法：第一阶段用单样本强化学习对齐任务格式，第二阶段通过多数投票奖励对齐基准分布。该方法在无训练数据的领域专用基准上，使基础模型性能显著提升，揭示其真实能力被直接评估低估。对于推理任务，对齐后微调模型与基础模型差距基本消失，表明文献中广泛报道的增强多为任务熟悉度伪象，非推理能力本质提升。
原文摘要 · Abstract (English)
Direct evaluation of LLMs on benchmarks can be misleading because comparatively strong performance may reflect task familiarity rather than capability. The train-before-test approach controls for task familiarity by giving each model task-relevant training before evaluation, originally through supervised finetuning. However, suitable training data is often hard to come by, and evaluation results vary with the data chosen. In this paper, we propose a two-stage test-time reinforcement learning (RL) alignment method for train-before-test. First, RL with a single sample provides a first alignment of the model to the task format, and second, test-time RL with majority-voting reward aligns the model to the benchmark distribution. Our test-time RL alignment method aligns similarly well as SFT-based train-before test, but without requiring a task-specific training set. On a domain-specific benchmark without training data, we show that direct evaluation underestimates base models which perform substantially better once aligned, yielding a more faithful evaluation of their capabilities. Moreover, for reasoning tasks, the performance gap between fine-tuned models and their base models largely disappears after alignment, suggesting that many gains from RLVR/SFT reported in the literature are not a difference in reasoning capability, but rather artifacts of task familiarity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。