测试大模型在百万级上下文中的代码理解与修复能力，发现当前模型仍严重依赖上下文长度。
LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
- 基于真实GitHub问题构建代码问答与缺陷修复任务，模拟长上下文场景。
- 多模型测试显示性能随上下文增长显著下降，如Claude 3.5从29%降至3%。
- 适合评估大模型在实际编程任务中处理长文本的能力，尤其关注代码理解。
模型上下文长度近年从数千迅速扩展至百万级别，但极端长上下文使构建真实可靠的基准测试变得困难，不仅因收集百万级上下文任务成本高，也因难以识别真正需要长上下文的场景。本文将代码理解与修复视为长上下文模型的自然测试场，提出LongCodeBench（LCB）基准，用于评估大语言模型在长上下文下的编码能力。该基准通过真实GitHub问题构建代码问答（LongCodeQA）和缺陷修复（LongSWE-Bench）任务，覆盖不同复杂度层级，可评估从Qwen2.5 14B Instruct到Google旗舰模型Gemini等多规模模型。实验发现，所有模型在长上下文下表现均明显退化，如Claude 3.5 Sonnet从29%降至3%，Qwen2.5从70.2%降至40%。数据集公开于https://huggingface.co/datasets/Steefano/LCB，代码库见https://github.com/Zteefano/long-code-bench。
原文摘要 · Abstract (English)
Context lengths for models have grown rapidly, from thousands to millions of tokens in just a few years. The extreme context sizes of modern long-context models have made it difficult to construct realistic long-context benchmarks -- not only due to the cost of collecting million-context tasks but also in identifying realistic scenarios that require significant contexts. We identify code comprehension and repair as a natural testbed and challenge task for long-context models and introduce LongCodeBench (LCB), a benchmark to test LLM coding abilities in long-context scenarios. Our benchmark tests both the comprehension and repair capabilities of LCLMs in realistic and important settings by drawing from real-world GitHub issues and constructing QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. We carefully stratify the complexity of our benchmark, enabling us to evaluate models across different scales -- ranging from Qwen2.5 14B Instruct to Google's flagship Gemini model. We find that long-context remains a weakness for all models, with performance drops such as from 29% to 3% for Claude 3.5 Sonnet, or from 70.2% to 40% for Qwen2.5. The LCB dataset is available publicly at https://huggingface.co/datasets/Steefano/LCB and the codebase to replicate the work on this paper at https://github.com/Zteefano/long-code-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。