构建首个涵盖代码与非代码要素的软件问题定位基准数据集
A Benchmark for Localizing Code and Non-Code Issues in Software Projects
- 设计MULocBench，整合代码与非代码证据，覆盖46个热门项目
- 现有方法在文件级定位准确率不足40%，难以应对复杂问题
- 适合研究软件维护、智能编程助手及大模型提示工程的学者
精准定位软件项目中的问题（如文件和函数）是维护的关键第一步。然而，现有基准如SWE-Bench和LocBench主要关注拉取请求类问题和代码位置，忽略提交记录、评论、配置和文档等非代码证据。为此，我们引入MULocBench，一个包含46个流行GitHub Python项目中1,100个问题的综合性数据集。相比已有基准，MULocBench在问题类型、根本原因、定位范围和文件类型上更具多样性，提供更贴近现实的评估环境。我们使用该基准评估了前沿定位方法及五种LLM提示策略，结果表明当前技术存在显著局限：即使在文件级别，准确率（Acc@5）和F1值均低于40%。这凸显了在多维度真实问题中实现泛化定位的挑战。为推动未来研究，我们已将MULocBench公开于https://huggingface.co/datasets/somethingone/MULocBench。
原文摘要 · Abstract (English)
Accurate project localization (e.g., files and functions) for issue resolution is a critical first step in software maintenance. However, existing benchmarks for issue localization, such as SWE-Bench and LocBench, are limited. They focus predominantly on pull-request issues and code locations, ignoring other evidence and non-code files such as commits, comments, configurations, and documentation. To address this gap, we introduce MULocBench, a comprehensive dataset of 1,100 issues from 46 popular GitHub Python projects. Comparing with existing benchmarks, MULocBench offers greater diversity in issue types, root causes, location scopes, and file types, providing a more realistic testbed for evaluation. Using this benchmark, we assess the performance of state-of-the-art localization methods and five LLM-based prompting strategies. Our results reveal significant limitations in current techniques: even at the file level, performance metrics (Acc@5, F1) remain below 40%. This underscores the challenge of generalizing to realistic, multi-faceted issue resolution. To enable future research on project localization for issue resolution, we publicly release MULocBench at https://huggingface.co/datasets/somethingone/MULocBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。