用自动化方法从已有题目生成开放性编程题,提升大模型解题能力。
FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale

- 基于竞赛题迭代演化出开放性编程问题,通过多样性指标筛选优质变体。
- 在两个基准上训练后,大模型得分提升超8分,Elo评分飙升300+。
- 生成题更接近人类设计,适合训练长流程、高复杂度的代码模型。
许多现实编程挑战是开放性的,没有已知最优解。然而,当前大语言模型编码研究主要集中在定义明确的任务上,如功能实现、修复漏洞和编程竞赛。开放性编程仍是大模型的短板,主要因为这类训练题稀缺且构建成本高。本文提出FrontierSmith,一个可规模化生成开放性编程题的自动化系统。该系统从现有封闭式编程题出发,通过改变目标、限制输出、泛化输入等方式生成候选开放题,并利用量化想法分歧度指标,筛选出能激发不同求解者产生真正多样化思路的问题。随后,智能体为保留下来的题目生成测试用例与验证器。在两个开放性编程基准上,使用合成数据训练后,模型性能显著提升:Qwen3.5-9B在FrontierCS上得分提高8.82,在ALE-bench上以Elo评分提升306.36;Qwen3.5-27B分别提升12.12和309.12。合成题目还促使智能体执行更多步骤、使用更多词元,与人工标注题表现相似,表明封闭题可作为长周期编码数据的有效起点。
原文摘要 · Abstract (English)
Many real-world coding challenges are open-ended and admit no known optimal solution. Yet, recent progress in LLM coding has focused on well-defined tasks such as feature implementation, bug fixing, and competitive programming. Open-ended coding remains a weak spot for LLMs, largely because open-ended training problems are scarce and expensive to construct. Our goal is to synthesize open-ended coding problems at scale to train stronger LLM coders. We introduce FrontierSmith, an automated system for iteratively evolving open-ended problems from existing closed-ended coding tasks. Starting from competitive programming problems, FrontierSmith generates candidate open-ended variants by changing the problems'goals, restricting outputs, and generalizing inputs. It then uses a quantitative idea divergence metric to select problems that elicit genuinely diverse approaches from different solvers. Agents then generate test cases and verifiers for the surviving candidates. On two open-ended coding benchmarks, training on our synthesized data yields substantial gains over the base models: Qwen3.5-9B improves by +8.82 score on FrontierCS and +306.36 (Elo-rating-based performance) on ALE-bench; Qwen3.5-27B improves by +12.12 and +309.12, respectively. The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones, suggesting that closed-ended seeds can be a practical starting point for long-horizon coding data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。