让大模型在测试时自动生成难题来练推理,边学边进化。
TTCS: Test-Time Curriculum Synthesis for Self-Evolving
- 测试时通过自动生成递进难题构建学习课程
- 在数学基准上显著提升推理能力,跨模型通用
- 适合想让大模型持续自我优化的研究者
测试时训练为仅用测试题改进大语言模型的推理能力提供了新路径。然而,现有方法在处理复杂推理题时表现不佳,原因有二:原始测试题过于困难,难以生成高质量伪标签;测试集规模有限,导致持续在线更新易引发不稳定。为此,我们提出TTCS,一种协同进化的测试时训练框架。具体而言,TTCS从同一预训练模型中初始化两个策略:问题生成器和推理求解器。二者通过迭代优化共同演化:生成器基于测试题生成逐步变难的问题变体,构建适配求解器当前能力的结构化课程;求解器则利用多轮采样响应的自一致性奖励,在原始题与合成题上进行自身更新。关键在于,求解器的反馈引导生成器产出更匹配当前能力的问题,而生成的问题变体反过来稳定求解器的测试时训练。实验表明,TTCS在挑战性数学基准上持续增强推理能力,并可迁移至不同骨干模型的通用任务,展示了动态构建测试时课程以实现自进化的大规模可行路径。代码与实现细节见 https://github.com/XMUDeepLIT/TTCS。
原文摘要 · Abstract (English)
Test-Time Training offers a promising way to improve the reasoning ability of large language models (LLMs) by adapting the model using only the test questions. However, existing methods struggle with difficult reasoning problems for two reasons: raw test questions are often too difficult to yield high-quality pseudo-labels, and the limited size of test sets makes continuous online updates prone to instability. To address these limitations, we propose TTCS, a co-evolving test-time training framework. Specifically, TTCS initializes two policies from the same pretrained model: a question synthesizer and a reasoning solver. These policies evolve through iterative optimization: the synthesizer generates progressively challenging question variants conditioned on the test questions, creating a structured curriculum tailored to the solver's current capability, while the solver updates itself using self-consistency rewards computed from multiple sampled responses on both original test and synthetic questions. Crucially, the solver's feedback guides the synthesizer to generate questions aligned with the model's current capability, and the generated question variants in turn stabilize the solver's test-time training. Experiments show that TTCS consistently strengthens the reasoning ability on challenging mathematical benchmarks and transfers to general-domain tasks across different LLM backbones, highlighting a scalable path towards dynamically constructing test-time curricula for self-evolving. Our code and implementation details are available at https://github.com/XMUDeepLIT/TTCS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。