构建复杂代码理解测试集，评估模型跨段落推理能力。
SWE-QA: A Dataset and Benchmark for Complex Code Understanding
- 基于12个Python仓库生成9072道多跳选择题，覆盖定义调用、组件交互等模式。
- 顶尖模型准确率仅74.41%，显示当前大模型在代码跨段推理上仍有明显短板。
- 适合关注代码智能、编程辅助和大模型推理能力的研究者使用。
本文提出SWE-QA，一个用于评估多跳代码理解的文本与代码语料库，弥补了简化评测任务与真实软件开发中复杂推理需求之间的差距。现有代码理解基准多聚焦孤立代码片段，而开发者需频繁连接分散代码段中的信息。该数据集包含9,072道多项选择题，系统性地源自SWE-bench的12个Python仓库，涵盖声明-调用类问题（链接实体定义与使用）及交互实体类问题（考察多个协作组件间的动态关系）。题目通过基于解析的实体提取与大语言模型辅助生成，并经仔细验证的干扰项确保真正理解而非表面匹配。对15个语言模型（参数量360M至671B）的评估显示，多跳推理仍具显著挑战，最佳表现达74.41%准确率。密集架构始终比混合专家模型高10-14个百分点，而增强推理的变体效果不一致。
原文摘要 · Abstract (English)
In this paper, we introduce SWE-QA, a text and code corpus aimed at benchmarking multi-hop code comprehension, addressing the gap between simplified evaluation tasks and the complex reasoning required in real-world software development. While existing code understanding benchmarks focus on isolated snippets, developers must routinely connect information across multiple dispersed code segments. The dataset comprises 9,072 multiple-choice questions systematically generated from 12 Python repositories of SWE-bench, evaluating several recurrent reasoning patterns like Declaration-and-Call questions that link entity definitions to their usage, and Interacting-Entity questions that examine the dynamic relationships among multiple collaborating components. Generated through parsing-based entity extraction and Large Language Model assisted question construction with carefully validated distractors, the benchmark distinguishes genuine comprehension from superficial pattern matching. Evaluation of 15 language models (360M to 671B parameters) reveals significant challenges in multi-hop reasoning, with best performance reaching 74.41% accuracy. Dense architectures consistently outperform mixture-of-experts models by 10-14 percentage points, while reasoning-enhanced variants show inconsistent benefits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。