让大模型生成能复现代码库功能的最小自包含文件，测试其真实理解能力。
Gistify! Codebase-Level Understanding via Runtime Execution
- 通过运行时执行追踪，要求模型提炼代码库核心功能
- 当前最强模型在长执行路径任务上成功率不足50%
- 适合评估代码大模型在真实项目中的理解深度
随着编程代理在大型代码库中部署增多，自动设计具有挑战性的代码库级评估变得至关重要。我们提出 Gistify 任务：要求编码大模型创建一个单一、最小且自包含的文件，以重现代码库的特定功能。模型可访问完整代码库及指定入口点（如 Python 命令），生成文件需复现相同命令在完整代码库中的输出，同时仅包含执行该命令所必需的核心组件。成功完成 Gistify 需要对代码库结构有深入理解、准确建模执行流程，并能生成可能较大的代码补丁。我们的研究发现，当前最先进模型在处理长执行轨迹的任务时表现不佳，难以可靠解决。
原文摘要 · Abstract (English)
As coding agents are increasingly deployed in large codebases, the need to automatically design challenging, codebase-level evaluation is central. We propose Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality of a codebase. The coding LLM is given full access to a codebase along with a specific entrypoint (e.g., a python command), and the generated file must replicate the output of the same command ran under the full codebase, while containing only the essential components necessary to execute the provided command. Success on Gistify requires both structural understanding of the codebase, accurate modeling of its execution flow as well as the ability to produce potentially large code patches. Our findings show that current state-of-the-art models struggle to reliably solve Gistify tasks, especially ones with long executions traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。