arXiv:2606.09052cs.LGcs.AI2026-06

让模型自动生成题目并自我进化,提升推理能力。

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

论文配图:INFUSER: Influence-Guided Self-Evolution Improves Reasoning
图 1 · 摘自论文原文
  • 生成器与求解器协同进化,动态优化题目质量。
  • 在奥数和SuperGPQA上相对基线提升超20%。
  • 适合想提升模型推理能力的研究者和开发者。

自进化为强化语言模型推理能力提供了可扩展路径:仅需少量外部监督即可使预训练模型自我改进。现有方法要么依赖大量人工或教师生成的数据,要么在无监督生成时使用难以提升求解能力的难度启发式奖励。我们提出INFUSER,一种迭代式联合训练框架,包含两个协同进化的角色:生成器从非结构化文档池中生成问题与标准答案,求解器则基于这些数据进行训练。求解器通过标准正确性奖励学习,而生成器则依据一种优化器感知的影响得分进行奖励,该得分衡量每个生成的问题是否真能改善求解器在目标分布上的表现。由于该连续且嘈杂的影响得分不适用于标准GRPO,我们提出DuGRPO——一种双归一化版本的GRPO,用于生成器训练。由此,文档池演化为自适应课程,优先选择对当前求解器有帮助的问题,而非仅难的问题。在Qwen3-8B-Base上,INFUSER在奥数和SuperGPQA基准上相较强自进化基线实现超过20%的相对提升;一个8B的共进化生成器在数学和编码任务上超越了冻结的32B思维生成器。消融实验验证了各项设计必要性,两个扩展——应用于指令微调锚点及引入规则可验证的RLVR数据——进一步证明了框架的灵活性与泛化能力。代码已开源于https://github.com/FFishy-git/INFUSER。

原文摘要 · Abstract (English)

Self-evolution offers a scalable path to stronger reasoning: a pretrained language model improves itself with only minimal external supervision. Yet existing methods either depend on extensively curated or teacher-generated training data, or, when the generator runs unsupervised, reward it by a difficulty heuristic that need not improve the solver. We introduce INFUSER, an iterative co-training framework with two co-evolving roles: a Generator that drafts questions and reference golden answers from a pool of unstructured, automatically collected documents, and a Solver that improves by training on them. The solver is trained with standard correctness rewards against the generator-provided answers, while the generator is rewarded by an optimizer-aware influence score that measures whether each proposed question would actually improve the solver on the target distribution. Because this continuous, noisy influence score is poorly served by standard GRPO, we propose DuGRPO, a dual-normalized variant of GRPO, for generator training. Together, these turn the document pool into an adaptive curriculum that favors questions useful to the current solver, not just hard ones. On Qwen3-8B-Base, INFUSER outperforms strong self-evolution baselines with over 20% relative improvement on Olympiad and SuperGPQA benchmarks, and an 8B INFUSER co-evolving generator outperforms a frozen 32B thinking generator on math and coding. Ablations confirm each design choice is necessary, and two extensions, applying INFUSER to an instruction-finetuned anchor and augmenting it with rule-verifiable RLVR data, further demonstrate the flexibility and generalizability of the framework. Code is available at https://github.com/FFishy-git/INFUSER.

自进化推理增强协同训练强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。