用真实难题引导大模型自博弈，提升代码生成能力
GASP: Guided Asymmetric Self-Play For Coding LLMs
- 以真实难题为靶心，逐步构建由易到难的训练题库
- 在LiveCodeBench上比无引导方法提升2.5%通过率
- 适合想提升代码模型泛化能力的研究者
非对称自博弈已成为大语言模型后训练的有前景范式，其中教师持续生成学生可解决但具挑战性的问题。然而，当前方法缺乏目标导向，无法区分真正有助于模型能力提升的问题。本文提出受引导的非对称自博弈（GASP），引入真实数据中的目标难题作为探索基准。训练中，教师先生成较易变体，再生成更难变体，逐步逼近目标难题。该方法使LiveCodeBench（LCB）上pass@20提升2.5%，并成功解决了所有基线无法触及的高难度目标问题。
原文摘要 · Abstract (English)
Asymmetric self-play has emerged as a promising paradigm for post-training large language models, where a teacher continually generates questions for a student to solve at the edge of the student's learnability. Although these methods promise open-ended data generation bootstrapped from no human data, they suffer from one major problem: not all problems that are hard to solve are interesting or informative to improve the overall capabilities of the model. Current asymmetric self-play methods are goal-agnostic with no real grounding. We propose Guided Asymmetric Self-Play (GASP), where grounding is provided by real-data goalpost questions that are identified to pose a hard exploration challenge to the model. During self-play, the teacher first generates an easier variant of a hard question, and then a harder variant of that easier question, with the goal of gradually closing the gap to the goalpost throughout training. Doing so, we improve pass@20 on LiveCodeBench (LCB) by 2.5% over unguided asymmetric self-play, and through the curriculum constructed by the teacher, we manage to solve hard goalpost questions that remain out of reach for all baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。