提出新基准与数据集，评估模型在代码仓库中修改代码的能力。
Turning the Tide: Repository-based Code Reflection
- 构建多文件仓库场景下的代码修改评测基准
- 涵盖6种语言1888个高难度测试用例，避免数据污染
- 适合研究代码生成与智能修复的开发者和研究人员
代码大模型通过理解与生成跨语言代码，提升编程效率与可访问性，支持智能反馈、错误检测与代码更新。现有基准（如HumanEval/LiveCodeBench）聚焦代码生成与真实场景相关性，但忽视了在代码仓库中修改代码的场景。针对反射能力提升与动态基准中的数据污染问题，本文提出LiveRepoReflection——一个面向多文件仓库上下文的挑战性评测基准，包含6种编程语言、1888个严格筛选的测试用例，确保多样性、正确性与高难度。同时，构建RepoReflection-Instruct数据集，基于多样化来源进行质量过滤，用于训练采用两轮对话（代码生成+错误驱动修复）的RepoReflectionCoder模型。排行榜对40多个大模型进行了评估，反映其在仓库级代码反思能力的表现。
原文摘要 · Abstract (English)
Code large language models (LLMs) enhance programming by understanding and generating code across languages, offering intelligent feedback, bug detection, and code updates through reflection, improving development efficiency and accessibility. While benchmarks (e.g. HumanEval/LiveCodeBench) evaluate code generation and real-world relevance, previous works ignore the scenario of modifying code in repositories. Considering challenges remaining in improving reflection capabilities and avoiding data contamination in dynamic benchmarks, we introduce LiveRepoReflection, a challenging benchmark for evaluating code understanding and generation in multi-file repository contexts, featuring 1,888 rigorously filtered test cases across $6$ programming languages to ensure diversity, correctness, and high difficulty. Further, we create RepoReflection-Instruct, a large-scale, quality-filtered instruction-tuning dataset derived from diverse sources, used to train RepoReflectionCoder through a two-turn dialogue process involving code generation and error-driven repair. The leaderboard evaluates over 40 LLMs to reflect the model performance of repository-based code reflection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。