用新方法让大模型自动生成难解编程题,提升推理能力。
QueST: Incentivizing LLMs to Generate Difficult Problems
- 通过难度感知采样与拒收微调,训练生成器造难题。
- 8B模型用10万条自动生成难题后超越原版,媲美671B大模型。
- 适合想提升模型推理能力的研究者和竞赛选手。
大型语言模型在推理任务中表现优异,能解决竞赛级编程与数学问题。但其可扩展性受限于人工标注数据集规模及缺乏大规模、高难度的编程训练数据。现有竞赛数据集仅含数千至数万道题目。以往合成数据生成方法依赖现有指令数据增强或从人工标注数据中挑选难题。本文提出QueST框架,结合难度感知图采样与难度感知拒收微调,直接优化专用生成器以创造高难度编程题。训练后的生成器在生成挑战性题目方面优于GPT-4o。我们利用QueST生成大规模合成编程题,并用于从强教师模型(带长思维链)进行知识蒸馏,或对小模型实施强化学习,均取得显著效果。蒸馏实验显示:将Qwen3-8B-base在10万条由QueST生成的难题上微调后,其在LiveCodeBench上的表现超过原始版本;额外加入11.2万条样本(即2.8万道人工题配多个合成解法),8B模型性能达到远超自身的DeepSeek-R1-671B水平。结果表明,通过QueST生成复杂问题是一种高效且可扩展的推进大模型竞赛编程与推理能力的方法。
原文摘要 · Abstract (English)
Large Language Models have achieved strong performance on reasoning tasks, solving competition-level coding and math problems. However, their scalability is limited by human-labeled datasets and the lack of large-scale, challenging coding problem training data. Existing competitive coding datasets contain only thousands to tens of thousands of problems. Previous synthetic data generation methods rely on either augmenting existing instruction datasets or selecting challenging problems from human-labeled data. In this paper, we propose QueST, a novel framework which combines difficulty-aware graph sampling and difficulty-aware rejection fine-tuning that directly optimizes specialized generators to create challenging coding problems. Our trained generators demonstrate superior capability compared to even GPT-4o at creating challenging problems that benefit downstream performance. We leverage QueST to generate large-scale synthetic coding problems, which we then use to distill from strong teacher models with long chain-of-thought or to conduct reinforcement learning for smaller models, proving effective in both scenarios. Our distillation experiments demonstrate significant performance gains. Specifically, after fine-tuning Qwen3-8B-base on 100K difficult problems generated by QueST, we surpass the performance of the original Qwen3-8B on LiveCodeBench. With an additional 112K examples (i.e., 28K human-written problems paired with multiple synthetic solutions), our 8B model matches the performance of the much larger DeepSeek-R1-671B. These findings indicate that generating complex problems via QueST offers an effective and scalable approach to advancing the frontiers of competitive coding and reasoning for large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。