推理模型编码表现依赖精准指令理解，越难任务越暴露短板。
A Case Study of Web App Coding with OpenAI Reasoning Models
- 用o1系列模型在WebApp1K上测试，发现其性能随任务复杂度提升而下降
- 新基准WebApp1K-Duo任务量翻倍后，o1模型性能大幅下滑，落后于Claude 3.5
- 对非常规但正确的测试用例易失败，说明其依赖严格指令遵循能力
本文通过案例研究比较了OpenAI最新推理模型o1-preview与o1-mini在编码任务中的表现，相较于其他前沿模型，o1系列在WebApp1K单任务基准上取得最优结果。为此，我们提出更难的WebApp1K-Duo基准，任务数与测试用例数均翻倍。该基准下，o1模型性能显著下降，甚至落后于Claude 3.5。此外，当遇到非典型但正确的测试用例时，o1模型持续失败，而无推理能力的模型偶尔可避开此陷阱。我们推测性能波动源于指令理解差异：推理机制在完全理解需求时提升表现，一旦遗漏关键要求则放大错误，可能受输入长度影响。因此，推理模型的成功依赖于高质量基础模型与监督微调（SFT）以确保精确执行指令。
原文摘要 · Abstract (English)
This paper presents a case study of coding tasks by the latest reasoning models of OpenAI, i.e. o1-preview and o1-mini, in comparison with other frontier models. The o1 models deliver SOTA results for WebApp1K, a single-task benchmark. To this end, we introduce WebApp1K-Duo, a harder benchmark doubling number of tasks and test cases. The new benchmark causes the o1 model performances to decline significantly, falling behind Claude 3.5. Moreover, they consistently fail when confronted with atypical yet correct test cases, a trap non-reasoning models occasionally avoid. We hypothesize that the performance variability is due to instruction comprehension. Specifically, the reasoning mechanism boosts performance when all expectations are captured, meanwhile exacerbates errors when key expectations are missed, potentially impacted by input lengths. As such, we argue that the coding success of reasoning models hinges on the top-notch base model and SFT to ensure meticulous adherence to instructions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。