用大模型自动生成竞赛级编程题，准确率达99%
AutoCode: LLMs as Problem Setters for Competitive Programming
- 多轮验证机制生成符合竞赛标准的题目与测试用例
- 生成题目在未见数据上达99%判断一致性，远超现有方法
- 可生成新变体题，适合高水平编程竞赛选手使用
编写编程竞赛题目要求极高：需设定约束、输入分布和边界情况以排除投机取巧，精准匹配特定算法（如最大流、动态规划、数据结构），并控制难度超越多数参赛者。我们认为这正适合检验通用大语言模型的能力。本文提出AutoCode，通过多轮验证生成竞赛级题目与测试用例。在保留问题上，AutoCode测试集与官方判题一致率达99%，显著优于当前最佳方法HardTests（低于81%）。从随机种子题目出发，AutoCode可生成带参考解与暴力解的新变体题，通过交叉验证解法与测试用例，进一步过滤无效题目。经专家人工验证，系统输出具有高正确性，且由大师级（前0.3%）选手判定为具备比赛质量。
原文摘要 · Abstract (English)
Writing competitive programming problems is exacting. Authors must: set constraints, input distributions, and edge cases that rule out shortcuts; target specific algorithms (e.g., max-flow, dynamic programming, data structures); and calibrate complexity beyond the reach of most competitors. We argue that this makes for an ideal test of general large language model capabilities and study whether they can do this reliably. We introduce AutoCode, which uses multiple rounds of validation to yield competition-grade problem statements and test cases. On held-out problems, AutoCode test suites approach 99% consistency with official judgments, a significant improvement over current state-of-the-art methods like HardTests, which achieve less than 81%. Furthermore, starting with a random seed problem, AutoCode can create novel variants with reference and brute-force solutions. By cross-verifying these generated solutions against test cases, we can further filter out malformed problems. Our system ensures high correctness, as verified by human experts. AutoCode successfully produces novel problems judged by Grandmaster-level (top 0.3%) competitive programmers to be of contest quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。