构建首个面向代码仓库级问答的基准，测试大模型理解复杂项目的能力。
SWE-QA: Can Language Models Answer Repository-level Code Questions?
- 基于7.7万条GitHub问题，设计覆盖跨文件、多跳依赖的576个问答对。
- 提出SWE-QA-Agent框架，实现大模型自主推理与自动求解。
- 验证大模型在真实项目理解中的潜力，揭示当前技术短板。
理解与推理整个软件仓库是智能软件工程工具的核心能力。现有基准如CoSQA和CodeQA主要关注小型自包含代码片段，难以体现真实仓库的复杂性——有效理解常需跨多个文件、把握软件架构，并基于长距离代码依赖定位答案。本文提出SWE-QA，一个面向仓库级代码问答的基准，旨在推动真实代码环境中自动化问答系统的研究。SWE-QA包含576个高质量问答对，涵盖意图理解、跨文件推理与多跳依赖分析等多样类别。研究首先从11个流行仓库中爬取77,100条GitHub问题，基于开发者自然提问行为，建立两级仓库级问题分类体系，并生成每类种子问题。随后人工筛选并验证问题，收集对应答案。作为原型应用，我们进一步开发SWE-QA-Agent——一个由大语言模型代理驱动的推理与行动框架。我们在多种上下文增强策略下评估六种先进大模型在SWE-QA上的表现。实验表明，大模型（尤其是SWE-QA-Agent）在解决仓库级问答任务上展现出潜力，同时也暴露出关键挑战，指明未来研究方向。
原文摘要 · Abstract (English)
Understanding and reasoning about entire software repositories is an essential capability for intelligent software engineering tools. While existing benchmarks such as CoSQA and CodeQA have advanced the field, they predominantly focus on small, self-contained code snippets. These setups fail to capture the complexity of real-world repositories, where effective understanding and reasoning often require navigating multiple files, understanding software architecture, and grounding answers in long-range code dependencies. In this paper, we present SWE-QA, a repository-level code question answering (QA) benchmark designed to facilitate research on automated QA systems in realistic code environments. SWE-QA involves 576 high-quality question-answer pairs spanning diverse categories, including intention understanding, cross-file reasoning, and multi-hop dependency analysis. To construct SWE-QA, we first crawled 77,100 GitHub issues from 11 popular repositories. Based on an analysis of naturally occurring developer questions extracted from these issues, we developed a two-level taxonomy of repository-level questions and constructed a set of seed questions for each category. For each category, we manually curated and validated questions and collected their corresponding answers. As a prototype application, we further develop SWE-QA-Agent, an agentic framework in which LLM agents reason and act to find answers automatically. We evaluate six advanced LLMs on SWE-QA under various context augmentation strategies. Experimental results highlight the promise of LLMs, particularly our SWE-QA-Agent framework, in addressing repository-level QA, while also revealing open challenges and pointing to future research directions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。