LLM生成代码常违背开发者隐含意图，现有测试方法无法发现此问题。
DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?
- 提出意图违反率IVR，通过隐藏约束测试捕捉未明说的开发意图。
- GPT-4.1和Claude Sonnet在92%以上测试中通过显式测试，但意图违反率超50%。
- 发现模型行为呈双峰模式，说明意图违背具有系统性，非偶然错误。
大语言模型生成的代码在面对模糊提示时可能违背开发者隐含意图，但现有基准仅衡量代码是否通过显式测试。本文提出意图违反率（IVR），并构建了基于HumanEval+的49个问题的初步基准。每个问题从明确提示中移除隐含约束，并将其编码为隐藏约束测试。IVR衡量的是：通过显式测试但未通过隐藏约束测试的生成代码比例，以反映未明说的开发意图。评估Claude Sonnet 4.6和OpenAI GPT 4.1发现，两者在超过92%的显式测试中通过，但在超过一半的问题中违反了隐含意图（分别为54.5%和63.5%），且呈现一致的系统性双峰模式。结果表明，通过率严重高估了生成代码对开发者意图的符合程度。
原文摘要 · Abstract (English)
Code generated by LLMs can violate a developer's implicit intentions when given an ambiguous prompt, yet standard benchmarks measure only whether code passes its stated test. We introduce the Intent Violation Rate (IVR) and a 49-problem pilot benchmark derived from HumanEval+. Each problem strips implicit constraints from a clarified prompt and encodes them as hidden constraint tests. IVR measures the fraction of LLM-generated solutions that pass the stated (visible) tests yet fail hidden constraint tests that capture unstated intent. Evaluating Claude Sonnet 4.6 and OpenAI GPT 4.1, we find both pass over 92\% of stated tests yet violate intent in over half of problems (54.5\% and 63.5\%), following a systematic, bimodal pattern consistent across both models. Out findings indicate that pass rates overstate how well generated code reflects developer intent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。