用真实代码库瓶颈评估大模型在多目标下优化代码的能力
FormulaCode: Evaluating Agentic Optimization on Large Codebases
- 基于真实科学项目挖掘957个性能瓶颈，构建多目标评估基准
- 每任务平均有264.6个社区维护的测试用例，确保评估真实性
- 揭示当前大模型在跨文件、多目标优化上仍存在显著短板
大型语言模型（LLM）编程代理正向仓库级操作演进，亟需能评估其在真实约束下优化整个代码库能力的基准。现有代码基准大多依赖合成任务、二值正确性信号或单目标评估，难以全面衡量代理的综合优化行为。我们提出FormulaCode，一个针对大型真实代码库进行代理优化评估的基准，具备细粒度、多目标性能指标。FormulaCode包含从GitHub上科学Python仓库中挖掘的957个性能瓶颈，每个瓶颈均配有专家编写补丁，且平均每项任务对应264.6个由社区维护的性能测试工作负载，使LLM代理能在真实正确的性能约束下评估其整体优化能力。我们的评估表明，当前前沿的LLM代理在仓库级、多目标优化方面仍面临巨大挑战。
原文摘要 · Abstract (English)
Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints. Existing code benchmarks largely rely on synthetic tasks, binary correctness signals, or single-objective evaluation, limiting their ability to assess holistic optimization behavior. We introduce FormulaCode, a benchmark for evaluating agentic optimization on large, real-world codebases with fine-grained, multi-objective performance metrics. FormulaCode comprises 957 performance bottlenecks mined from scientific Python repositories on GitHub, each paired with expert-authored patches and, on average, 264.6 community-maintained performance workloads per task, enabling the holistic ability of LLM agents to optimize codebases under realistic correctness and performance constraints. Our evaluations reveal that repository-scale, multi-objective optimization remains a major challenge for frontier LLM agents. Project website at: https://formula-code.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。