用检索与多轮自省提升单智能体模型解编程难题能力
Can Multi-turn Self-refined Single Agent LMs with Retrieval Solve Hard Coding Problems?
- 结合多轮自检、反思与记忆检索提升推理
- 解决率从19.1%提升至42.2%（pass@1）
- 适合研究大模型算法思维与人机协作的学者
竞争性编程是人类最困难的任务之一，要求复杂算法设计、解谜能力和高效编码。本文构建了包含254道国际大学生程序设计竞赛（ICPC）题目的基准数据集ICPC，每道题均配有官方解析、参考代码及公开、高阶和隐藏测试。基于此，我们评估多种语言模型推理技术。零样本链式思考提示下，o1模型仅达19.1%的pass@1解决率；采用多轮自检、反思与情景记忆检索相结合的最佳方法，提升至42.2%。此外，通过人机协同实验发现，仅需少量特定指令，o1即可解决此前所有模型无法破解的18题中的17题。定量与定性结果共同表明，该方法正推动大模型向具备立足现实、想象与算法思维的方向迈进。代码与数据已开源。
原文摘要 · Abstract (English)
Among the hardest tasks for humans are those found in competitive programming where problems require sophisticated algorithmic thinking, puzzle solving, and the creation of effective code. As a domain to assess language models (LMs), it has not received enough attention, though. This study presents the ICPC benchmark, which consists of 254 international collegiate programming contest (ICPC) tasks. Each problem includes official analysis, reference code, and sample, high-quality unit, and hidden tests. We are able to develop and evaluate a variety of LM inference techniques for competitive programming with these resources. With zero-shot chain-of-thought prompting, we find that o1 only achieves a 19.1\% pass@1 solve rate. With our best inference technique, which combines multi-turn self-judge with reflection and retrieval over episodic information, raises this to 42.2\%. Furthermore, we conduct a new human-in-the-loop investigation to gain a deeper understanding of the remaining difficulties. Surprisingly, we discover that o1 can solve 17 out of 18 problems that were previously unsolvable by any model or technique with just a few specific instructions. A footstep toward LMs with grounded, imaginative, and algorithmic thinking is provided by our quantitative findings and qualitative research. We open-source our code and data at https://github.com/kraritt/zolve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。