评测大模型长链条推理能力，挑战高达百万词的复杂推理任务。
LongReasonArena: A Long Reasoning Benchmark for Large Language Models
- 设计多步算法任务，模拟检索与回溯等长推理过程。
- 最复杂任务需100万词推理，模型准确率最低仅7.5%。
- 适合评估模型在长推理场景下的真实能力，如科研与工程应用。
现有大语言模型（LLM）的长上下文评测主要关注长文本理解，忽视了长推理能力的评估。为填补这一空白，我们提出LongReasonArena，一个专门用于评估LLM长推理能力的基准。该基准要求模型通过执行多步算法解决难题，体现关键的长推理特征，如信息检索与回溯。通过控制输入，可任意扩展所需推理长度，最复杂任务达到100万词的推理量。大量实验表明，LongReasonArena对开源和闭源模型均构成显著挑战：例如，Deepseek-R1在该任务上仅取得7.5%的准确率。进一步分析发现，准确率随预期推理步数的对数呈线性下降。代码与数据已公开于https://github.com/LongReasonArena/LongReasonArena。
原文摘要 · Abstract (English)
Existing long-context benchmarks for Large Language Models (LLMs) focus on evaluating comprehension of long inputs, while overlooking the evaluation of long reasoning abilities. To address this gap, we introduce LongReasonArena, a benchmark specifically designed to assess the long reasoning capabilities of LLMs. Our tasks require models to solve problems by executing multi-step algorithms that reflect key aspects of long reasoning, such as retrieval and backtracking. By controlling the inputs, the required reasoning length can be arbitrarily scaled, reaching up to 1 million tokens of reasoning for the most challenging tasks. Extensive evaluation results demonstrate that LongReasonArena presents a significant challenge for both open-source and proprietary LLMs. For instance, Deepseek-R1 achieves only 7.5% accuracy on our task. Further analysis also reveals that the accuracy exhibits a linear decline with respect to the logarithm of the expected number of reasoning steps. Our code and data is available at https://github.com/LongReasonArena/LongReasonArena.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。