arXiv:2604.08468cs.LGcs.AI2026-04ACL被引 2

用测试时动态生成新问题,让大模型自我进化更强推理能力

TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis

  • 测试时自动生成语义等价的问题变体,避免学表面模式
  • 在8种模型上超越现有方法,甚至超过依赖大量标注数据的强基线
  • 适合需要快速适应新场景且无标注数据的智能系统

尽管基于可验证奖励的强化学习(RLVR)推动了大型推理模型(LRMs)的发展,但在专业或新领域中,此类监督成本过高或不可得,限制了测试时适应。现有测试时方法受限于静态查询集,易过拟合文本模式。为此,我们提出测试时变分合成(TTVS),使LRM通过从无标签测试查询中动态扩充训练流实现自我演化。TTVS包含两个协同模块:(1) 在线变分合成,将静态测试查询转化为多样、语义等价的动态变体,迫使模型学习底层问题逻辑而非表面模式;(2) 测试时混合探索,平衡精度驱动的利用与一致性驱动的探索。大量实验表明,TTVS在八种模型架构上均表现更优。值得注意的是,仅使用无标签测试数据,TTVS不仅优于其他测试时适配方法,还超越了依赖海量高质量标注数据的最先进监督式强化学习技术。

原文摘要 · Abstract (English)

Despite significant advances in Large Reasoning Models (LRMs) driven by reinforcement learning with verifiable rewards (RLVR), this paradigm is fundamentally limited in specialized or novel domains where such supervision is prohibitively expensive or unavailable, posing a key challenge for test-time adaptation. While existing test-time methods offer a potential solution, they are constrained by learning from static query sets, risking overfitting to textual patterns. To address this gap, we introduce Test-Time Variational Synthesis (TTVS), a novel framework that enables LRMs to self-evolve by dynamically augmenting the training stream from unlabeled test queries. TTVS comprises two synergistic modules: (1) Online Variational Synthesis, which transforms static test queries into a dynamic stream of diverse, semantically-equivalent variations, enforcing the model to learn underlying problem logic rather than superficial patterns; (2) Test-time Hybrid Exploration, which balances accuracy-driven exploitation with consistency-driven exploration across synthetic variants. Extensive experiments show TTVS yields superior performance across eight model architectures. Notably, using only unlabeled test-time data, TTVS not only surpasses other test-time adaptation methods but also outperforms state-of-the-art supervised RL-based techniques trained on vast, high-quality labeled data.

强化学习自进化测试时学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。