用自然语言稿子先解题再写代码，更真实评估大模型编程能力
Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming
- 先生成解题思路稿再写代码，提升模型解题成功率
- 专家级解题稿使解题率显著提升，但编码仍困难
- 提出新评测方法，适合研究模型推理与实现分离的能力
大型语言模型在编程竞赛中表现日益出色，但现有评估将算法推理与代码实现混为一谈。我们认为编程竞赛本质上是解题任务，主张以自然语言编辑稿为核心进行解法生成与评估。在编写代码前先生成编辑稿，可提升部分模型的解题率，尤其使用专家撰写的黄金编辑稿时提升更明显。然而，即使有黄金编辑稿，模型在实现阶段仍表现不佳；对比生成与黄金编辑稿的差异，揭示了算法描述不完整、不准确的持续性瓶颈。我们通过专家标注诊断推理错误，并验证了基于大模型自评的可扩展评估协议。引入一个包含83个ICPC风格题目的数据集，附带黄金编辑稿与完整测试套件，评估了19个大模型，主张未来基准应明确区分解题与实现。
原文摘要 · Abstract (English)
Large Language Models (LLMs) increasingly succeed on competitive programming problems, yet existing evaluations conflate algorithmic reasoning with code-level implementation. We argue that competitive programming is fundamentally a problem-solving task and propose centering natural-language editorials in both solution generation and evaluation. Generating an editorial prior to code improves solve rates for some LLMs, with substantially larger gains when using expertly written gold editorials. However, even with gold editorials, models continue to struggle with implementation, while the gap between generated and gold editorials reveals a persistent problem-solving bottleneck in specifying correct and complete algorithms. Beyond pass/fail metrics, we diagnose reasoning errors by comparing model-generated editorials to gold standards using expert annotations and validate an LLM-as-a-judge protocol for scalable evaluation. We introduce a dataset of 83 ICPC-style problems with gold editorials and full test suites, and evaluate 19 LLMs, arguing that future benchmarks should explicitly separate problem solving from implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。