现有评测只看测试通过率,忽略了代码设计合规性。
Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolution

- 构建新基准,显式量化项目设计约束
- 仅47%修复补丁完全符合设计规范
- 适合关注代码质量而非仅功能正确的研究者
仓库级问题修复评测已成为评估大模型代理的标准场景,但成功率仍主要以测试通过率衡量。实际上,可接受的补丁还需符合项目特定的设计约束,如架构惯例、错误处理策略和可维护性要求,这些常未被测试覆盖,仅隐含于代码评审讨论中。本文提出“设计感知问题修复”并构建ench{}基准,通过挖掘真实拉取请求中的设计约束,将其与问题实例关联,并用基于LLM的验证器自动检查补丁合规性,涵盖6个仓库的495个问题和1,787条已验证约束,与SWE-bench-Verified和SWE-bench-Pro对齐。实验表明,基于测试的正确性严重高估补丁质量:不足一半修复满足设计要求,设计违规普遍存在,且功能正确性与设计满足度无显著统计关联。即使提供具体设计指导,仍存在大量不合规现象,暴露出当前代理能力的根本短板,亟需超越功能正确性的设计感知评估。
原文摘要 · Abstract (English)
Repository-level issue resolution benchmarks have become a standard testbed for evaluating LLM-based agents, yet success is still predominantly measured by test pass rates. In practice, however, acceptable patches must also comply with project-specific design constraints, such as architectural conventions, error-handling policies, and maintainability requirements, which are rarely encoded in tests and are often documented only implicitly in code review discussions. This paper introduces \textit{design-aware issue resolution} and presents \bench{}, a benchmark that makes such implicit design constraints explicit and measurable. \bench{} is constructed by mining and validating design constraints from real-world pull requests, linking them to issue instances, and automatically checking patch compliance using an LLM-based verifier, yielding 495 issues and 1,787 validated constraints across six repositories, aligned with SWE-bench-Verified and SWE-bench-Pro. Experiments with state-of-the-art agents show that test-based correctness substantially overestimates patch quality: fewer than half of resolved issues are fully design-satisfying, design violations are widespread, and functional correctness exhibits negligible statistical association with design satisfaction. While providing issue-specific design guidance reduces violations, substantial non-compliance remains, highlighting a fundamental gap in current agent capabilities and motivating design-aware evaluation beyond functional correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。