arXiv:2509.21028cs.AI2025-09

构建科学长文本数值推理数据集,评估并提升大模型的复杂计算能力。

SciTrek: Evaluating and Improving Long-Context Numerical Reasoning over Scientific Articles

  • 用数据库元数据生成带真实数学操作的问答,逻辑透明可验证。
  • 128K上下文下最佳模型仅46.5%准确率,越长文本性能越差。
  • 适合研究长文本推理、模型微调与数学能力增强的开发者。

我们提出SciTrek,一个用于评估和改进大语言模型(LLM)长上下文数值推理能力的合成问答数据集。现有超过64K token的长上下文数据集要么仅用于简单信息检索,要么虽涉及推理但依赖人工构造的上下文,且数值推理普遍被忽视。SciTrek通过要求对全文科学文章集合进行计数、排序、聚合和比较等数值操作来填补这一空白。问题由文章元数据(标题、作者、参考文献)的SQL查询自动生成,真值答案通过执行这些查询获得。底层SQL提供可解释的推理规范,支持细粒度错误分析。此外,该全自动化流水线可无监督扩展至任意上下文长度和数据规模,减少数据污染,为后训练提供充足数据。大量实验表明,前沿开源与专有模型在看似简单的问题上表现不佳:最佳模型在128K token下仅达46.5%精确匹配,且性能随上下文增长持续下降。细粒度分析揭示了引用相关问题及复合逻辑条件(尤其含否定)上的系统性弱点。最后,基于SciTrek对开源模型进行后训练,可显著提升其数值推理能力,并泛化至非领域长上下文任务。

原文摘要 · Abstract (English)

We introduce SciTrek, a synthetic question-answering dataset for assessing and improving long-context numerical reasoning in large language models (LLMs). Existing long-context datasets with inputs beyond 64K tokens either target simple information retrieval or, when they do involve reasoning, rely on artificial contexts, while numerical reasoning remains largely overlooked in both cases. SciTrek addresses these limitations with questions that require numerical operations (e.g., counting, sorting, aggregation, and comparison) over collections of full-text scientific articles. Questions are generated automatically by formulating them as SQL queries over a database of article metadata (titles, authors, and references), and ground-truth answers are obtained by executing these queries. The underlying SQL provides a transparent, verifiable specification of the reasoning each question requires, enabling fine-grained error analysis. Furthermore, the proposed fully automated pipeline scales to arbitrary context lengths and dataset sizes with minimal human supervision, mitigating data contamination and supplying abundant data for post-training. Extensive experiments show that frontier open-weight and proprietary LLMs struggle even with ostensibly simple questions: the best-performing model achieves only 46.5% exact match at 128K tokens, and performance degrades steadily as contexts grow. Fine-grained analysis reveals systematic weaknesses on citation-related questions and on compound logical conditions, particularly those involving negation. Finally, post-training open-weight models on SciTrek improves their numerical reasoning in ways that generalise to out-of-domain long-context tasks.

长上下文数值推理科学问答数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。