发现代码评测基准中13.6%的请求与问题不匹配，提出自动检测工具。
PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
- 设计多智能体系统，分三阶段识别请求与问题的错配模式。
- 在两个基准上达到最高92.12%准确率，验证了检测可靠性。
- 适合关注代码生成评估质量的研究者和开发者使用。
SWE-bench类基准广泛用于评估大模型解决代码问题的能力。其通常采用固定流程：从请求（PR）描述中提取关联的问题引用，以问题描述作为任务说明，请求补丁作为测试标准。然而，由于大型仓库开发维护的复杂性，实际中请求与问题的配对常出现错位。本文系统研究SWE-bench Verified实例，发现13.6%存在五种模式下的十一类细粒度错配。为此，我们提出PAIChecker，一种用于检测SWE-bench类基准中请求-问题错配的多智能体系统。该系统采用三阶段设计：模式识别、跨智能体标签合成与代码级验证，实现更准确、可泛化且逐步验证的检测。在SWE-Gym和SWE-bench Multilingual上的实验表明，PAIChecker在所有四个大模型基座上表现最优，二分类准确率分别达到92.12%和91.67%。
原文摘要 · Abstract (English)
SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。