构建代码问答基准，区分真理解与文档记忆。
Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA
- 先生成答案再构造问题，确保题目基于真实代码结构。
- 代码访问带来0.23的准确率提升，文档仅增0.071收益。
- 适合评估模型代码理解能力，尤其关注免记忆测试。
我们提出Code-QA-Bench，一个全自动的仓库级代码理解基准生成框架，旨在分离真正的代码理解与文档记忆、预训练记忆。该框架包含两项方法贡献：(1) 答案优先生成流程，工具增强的智能体先探索源码生成可验证的正确答案，再推导问题，确保每道题均基于真实代码结构；(2) 三条件实验设计，在闭卷（无仓库）、仅代码（移除文档）和完整文档（全仓库）条件下评估智能体，通过差异量化文档效用与记忆程度。我们在SWE-Bench的10个Python仓库中生成528个代码可推导任务和100个依赖文档的任务，由LLM裁判在准确性、完整性与具体性上评分。对四个前沿模型的实验表明，代码访问是主要因素（相比闭卷平均提升0.23），文档提供小幅额外收益（在依赖文档任务上+0.071），且在代码可推导任务上，仅代码 ≈ 完整文档，验证了设计有效性。框架开源，适用于任何良好文档化的Python仓库。
原文摘要 · Abstract (English)
We present Code-QA-Bench, a fully automated framework for synthesizing repository-level code understanding benchmarks that separates genuine code comprehension from documentation recall and pretraining memorization. The framework makes two methodological contributions: (1) an answer-first generation pipeline where a tool-equipped agent explores source code to produce verified gold answers before deriving questions, ensuring every task is grounded in real code structure; and (2) a three-condition experimental design evaluating agents under closed-book (no repository), code-only (documentation removed), and documented (full repository) conditions, with deltas directly quantifying documentation utility and memorization. We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories from SWE-Bench, scored by an LLM judge on accuracy, completeness, and specificity. Experiments on four frontier models reveal that code access is the dominant factor (+0.23 mean gain over closed-book), documentation provides modest additional benefit (+0.071 on doc-dependent tasks), and code-only $\approx$ documented on code-derivable tasks, validating the design. The framework is open-source and applicable to any well-documented Python repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。