arXiv:2506.11066cs.SEcs.AI2025-06ACL被引 6

首个多语言代码质量评估基准,提升检索系统对安全与高效代码的识别能力。

CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval

  • 构建覆盖11语言的42725个查询与13万代码片段的质量标注数据集。
  • 发现顶尖模型仍常混淆有缺陷与高质量代码,准确率不足预期。
  • 引入新评估指标,适合关注代码安全与可维护性的研发团队使用。

代码检索在现代软件开发中至关重要,有助于代码复用与快速调试。然而,现有基准主要关注功能相关性,忽视了代码质量的关键维度。为此,我们提出CoQuIR,首个大规模、多语言的代码质量感知检索评估基准,涵盖正确性、效率、安全性和可维护性四大维度。CoQuIR为42,725个查询和134,907个代码片段提供了细粒度质量标注,支持11种编程语言,并引入两项以质量为核心的评估指标:成对偏好准确率(Pairwise Preference Accuracy)与基于边距的排序得分(Margin-based Ranking Score)。我们对23种检索模型(包括开源与专有系统)进行了基准测试,发现即使表现最优的模型也频繁无法区分有缺陷或不安全的代码与更稳健的版本。此外,我们通过合成数据集探索了显式鼓励质量识别的训练方法,在不牺牲语义相关性的前提下,显著提升了多个模型在质量感知指标上的表现。下游代码生成实验进一步验证了该方法的有效性。本研究强调将质量信号融入代码检索系统的重要性,为构建更可信、更鲁棒的软件开发工具奠定基础。

原文摘要 · Abstract (English)

Code retrieval is essential in modern software development, as it boosts code reuse and accelerates debugging. However, current benchmarks primarily emphasize functional relevance while neglecting critical dimensions of software quality. Motivated by this gap, we introduce CoQuIR, the first large-scale, multilingual benchmark specifically designed to evaluate quality-aware code retrieval across four key dimensions: correctness, efficiency, security, and maintainability. CoQuIR provides fine-grained quality annotations for 42,725 queries and 134,907 code snippets in 11 programming languages, and is accompanied by two quality-centric evaluation metrics: Pairwise Preference Accuracy and Margin-based Ranking Score. Using CoQuIR, we benchmark 23 retrieval models, covering both open-source and proprietary systems, and find that even top-performing models frequently fail to distinguish buggy or insecure code from their more robust counterparts. Furthermore, we conduct preliminary investigations into training methods that explicitly encourage retrievers to recognize code quality. Using synthetic datasets, we demonstrate promising improvements in quality-aware metrics across various models, without sacrificing semantic relevance. Downstream code generation experiments further validate the effectiveness of our approach. Overall, our work highlights the importance of integrating quality signals into code retrieval systems, laying the groundwork for more trustworthy and robust software development tools.

代码检索质量评估多语言基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。