用轻量探针加速任务生成,让模型在恰到好处的难度上训练。
Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier

- 训练任务生成器时引入可学习探针,替代耗时的求解器回滚
- 代码任务生成成功率从10.1%提升至20.0%,软件工程任务从9.8%升至19.6%
- 适合需要高效生成高价值训练任务的研究者和工业应用
强化学习训练智能体的瓶颈正日益来自高质量任务供给不足:即有效且可解、难度适中的任务。随着推理与智能体模型能力提升,固定任务分布趋于饱和,而简单合成任务常过于简单、无法求解或定义不清。直接使用强化学习优化任务生成器虽可缓解此问题,但需对每个候选任务反复调用求解器,成本极高——以软件工程任务为例,单次求解需数十分钟,导致求解器闭环训练不可行。本文提出PROPEL框架,通过求解器摊销策略实现目标求解率下的任务生成器训练。该方法仅需一次标注的任务-求解结果语料库,训练一个轻量级激活探针,预测由冻结生成器参考模型产生的任务在目标求解率下的通过率,并作为求解率代理用于生成器优化,使评估降至单次前向传播。在数学、代码与软件工程任务中,多个模型规模下验证均表明,该方法成功将生成任务推向目标求解率:代码任务生成在Qwen2.5-3B-Instruct模型上通过率从10.1%提升至20.0%,在Qwen2.5-7B-Instruct上从5.3%升至12.6%;软件工程任务在未见仓库上的目标求解率占比从9.8%增至19.6%。
原文摘要 · Abstract (English)
The limiting resource for training agents via reinforcement learning (RL) is increasingly frontier task supply: valid, solvable tasks just difficult enough to train the current model. As reasoning and agentic models improve, fixed task distributions saturate, while naive synthetic generation yields tasks that are trivial, impossible, or ill-posed. Training a task generator with RL to optimize validity and learnability can address this bottleneck, but direct optimization requires repeated solver rollouts per candidate. For software-engineering (SWE) tasks, a single rollout can take tens of minutes; solver-in-the-loop generator training is intractable. We introduce PROPEL, a solver-amortized framework for training task generators at the targeted solve rate. PROPEL trains a lightweight activation probe on a one-time labeled corpus of generated tasks and solver outcomes. The probe predicts target-solver pass rate from a frozen generator reference model and serves as a proxy for solve rate during generator optimization, reducing generator evaluation to a single forward pass. Across math, code, and software-engineering at multiple model scales, PROPEL shifts generation toward the targeted solve rate: for coding, tasks generated at the learnable frontier increase from $10.1\% \rightarrow 20.0\%$ for a Qwen2.5-3B-Instruct solver and from $5.3\% \rightarrow 12.6\%$ for a Qwen2.5-7B-Instruct solver. For SWE, PROPEL increases the share of generations at the targeted solve rate from $9.8\% \rightarrow 19.6\%$ for Qwen3.5-27B on repositories not seen during training of probe and generator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。