让大模型在推理时持续进化，突破性能瓶颈。
TEMPO: Scaling Test-time Training for Large Reasoning Models
- 交替优化策略与周期性校准评价器，防止奖励信号漂移
- 在AIME 2024上，7B模型准确率从33.0%提升至51.1%
- 适用于需要长期推理能力进化的大型语言模型
测试时训练（TTT）在推理阶段对未标注测试样本进行参数调整，持续拓展离线训练无法触及的能力。尽管初期有效，现有方法在大型推理模型上的表现迅速停滞，且无法利用额外的推理计算资源。缺乏外部校准导致自生成奖励信号随策略演化而漂移，引发性能瓶颈和多样性崩溃。本文提出TEMPO框架，通过在无标签问题上交替进行策略精炼与在有标签数据集上的周期性评判器重校准。基于期望最大化（EM）算法形式化该过程，揭示先前方法为不完整变体，缺失关键重校准步骤。重新引入该步骤可收紧证据下界（ELBO），实现持续改进。在多种模型家族（Qwen3 和 OLMO3）及推理任务中，TEMPO将OLMO3-7B在AIME 2024上的准确率从33.0%提升至51.1%，Qwen3-14B从42.3%提升至65.8%，同时保持高多样性。
原文摘要 · Abstract (English)
Test-time training (TTT) adapts model parameters on unlabeled test instances during inference time, which continuously extends capabilities beyond the reach of offline training. Despite initial gains, existing TTT methods for LRMs plateau quickly and do not benefit from additional test-time compute. Without external calibration, the self-generated reward signal increasingly drifts as the policy model evolves, leading to both performance plateaus and diversity collapse. We propose TEMPO, a TTT framework that interleaves policy refinement on unlabeled questions with periodic critic recalibration on a labeled dataset. By formalizing this alternating procedure through the Expectation-Maximization (EM) algorithm, we reveal that prior methods can be interpreted as incomplete variants that omit the crucial recalibration step. Reintroducing this step tightens the evidence lower bound (ELBO) and enables sustained improvement. Across diverse model families (Qwen3 and OLMO3) and reasoning tasks, TEMPO improves OLMO3-7B on AIME 2024 from 33.0% to 51.1% and Qwen3-14B from 42.3% to 65.8%, while maintaining high diversity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。