评测大模型在复杂软件开发中的长上下文理解能力，填补了真实场景评估空白。
LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering
- 构建覆盖10种语言的8000个任务，上下文长达10K到100万词元
- 提出8类核心任务和17项指标，量化长上下文性能退化问题
- 适合研究大模型代码理解、软件工程自动化与架构推理的开发者
长上下文大模型（上下文窗口达百万词元）为复杂代码理解和软件开发评估带来了新机遇。我们提出LoCoBench，一个专为真实复杂软件开发场景设计的综合性基准。不同于聚焦单函数补全或短上下文任务的现有基准，LoCoBench填补了对需理解整个代码库、跨文件推理及保持架构一致性的长上下文能力的评估空白。该基准涵盖10种编程语言，生成8000个评估场景，上下文长度覆盖10K至100万词元（100倍差异），可精准评估长上下文性能退化。引入8类任务：架构理解、跨文件重构、多会话开发、缺陷调查、功能实现、代码理解、集成测试与安全分析。通过五阶段流程生成多样化高质量场景，挑战模型在空前规模下的代码推理能力。提出包含17项指标（含8项新指标）的综合评估框架，计算洛科得分（LCBS）。对前沿长上下文模型的评估揭示显著性能差距，表明复杂软件开发中的长上下文理解仍是未解难题，亟需更多关注。项目开源地址：https://github.com/SalesforceAIResearch/LoCoBench。
原文摘要 · Abstract (English)
The emergence of long-context language models with context windows extending to millions of tokens has created new opportunities for sophisticated code understanding and software development evaluation. We propose LoCoBench, a comprehensive benchmark specifically designed to evaluate long-context LLMs in realistic, complex software development scenarios. Unlike existing code evaluation benchmarks that focus on single-function completion or short-context tasks, LoCoBench addresses the critical evaluation gap for long-context capabilities that require understanding entire codebases, reasoning across multiple files, and maintaining architectural consistency across large-scale software systems. Our benchmark provides 8,000 evaluation scenarios systematically generated across 10 programming languages, with context lengths spanning 10K to 1M tokens, a 100x variation that enables precise assessment of long-context performance degradation in realistic software development settings. LoCoBench introduces 8 task categories that capture essential long-context capabilities: architectural understanding, cross-file refactoring, multi-session development, bug investigation, feature implementation, code comprehension, integration testing, and security analysis. Through a 5-phase pipeline, we create diverse, high-quality scenarios that challenge LLMs to reason about complex codebases at unprecedented scale. We introduce a comprehensive evaluation framework with 17 metrics across 4 dimensions, including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our evaluation of state-of-the-art long-context models reveals substantial performance gaps, demonstrating that long-context understanding in complex software development represents a significant unsolved challenge that demands more attention. LoCoBench is released at: https://github.com/SalesforceAIResearch/LoCoBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。