构建更公平的代码理解评测基准,让模型真正懂代码而非靠记忆。
SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding
- 从长尾仓库构建评测数据集,避免模型靠记忆作弊。
- 用代理式流程解题比直接回答高13分,证明探索必要性。
- 开源小模型经训练后超越GPT-4o,适合研究智能编程工具者。
代理式仓库级代码理解对自动化复杂软件工程任务至关重要,但当前领域缺乏可靠评测基准。现有评估常忽略长尾主题,且依赖热门仓库,使大语言模型(LLMs)可通过记忆知识作弊。为此,我们提出SWE-QA-Pro,一个基于多样、长尾仓库并具备可执行环境的基准。通过问题驱动聚类实现主题平衡,覆盖欠代表的任务类型,并进行严格难度校准:过滤掉仅靠直接回答可解的问题。结果表明,代理工作流显著优于直接回答(如Claude Sonnet 4.5相差约13分),证实了代理式代码库探索的必要性。此外,为应对复杂行为训练数据稀缺问题,我们提出可扩展的合成数据流水线,支持两阶段训练流程:监督微调(SFT)后接基于AI反馈的强化学习(RLAIF)。该方法使小型开源模型学会高效使用工具与推理。实验证明,使用该流程训练的Qwen3-8B模型在SWE-QA-Pro上比GPT-4o高2.3分,大幅缩小与顶尖闭源模型的差距,验证了评测的有效性与训练流程的优越性。
原文摘要 · Abstract (English)
Agentic repository-level code understanding is essential for automating complex software engineering tasks, yet the field lacks reliable benchmarks. Existing evaluations often overlook the long tail topics and rely on popular repositories where Large Language Models (LLMs) can cheat via memorized knowledge. To address this, we introduce SWE-QA-Pro, a benchmark constructed from diverse, long-tail repositories with executable environments. We enforce topical balance via issue-driven clustering to cover under-represented task types and apply a rigorous difficulty calibration process: questions solvable by direct-answer baselines are filtered out. This results in a dataset where agentic workflows significantly outperform direct answering (e.g., a ~13-point gap for Claude Sonnet 4.5), confirming the necessity of agentic codebase exploration. Furthermore, to tackle the scarcity of training data for such complex behaviors, we propose a scalable synthetic data pipeline that powers a two-stage training recipe: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from AI Feedback (RLAIF). This approach allows small open models to learn efficient tool usage and reasoning. Empirically, a Qwen3-8B model trained with our recipe surpasses GPT-4o by 2.3 points on SWE-QA-Pro and substantially narrows the gap to state-of-the-art proprietary models, demonstrating both the validity of our evaluation and the effectiveness of our agentic training workflow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。