用结构化表格数据提升大模型长上下文推理能力
Probing How Scalable Table Data Enhances General Long-Context Reasoning
- 通过分析表格数据的周期性依赖结构，发现其适合长上下文推理
- 合成表格数据使模型在多个基准上平均提升8.24%的推理能力
- 方法简单可扩展，适合用于大模型后训练数据增强
随着现实任务日益复杂，长上下文推理已成为大语言模型（LLMs）的核心能力。然而，少有研究探讨何种数据类型对长上下文推理有效及其原因。我们发现具有周期结构的结构化表格数据在长上下文推理中表现出显著潜力。基于此，我们利用互信息数学分析表格依赖结构，揭示了表格数据中非衰减的周期性依赖关系。进一步系统评估了结构化表格数据的能力，开展规模化实验并验证其增强长上下文推理的机制，获得若干有意义发现。据此，我们提出一个简单且可扩展的流水线（TableLong），用于生成高质量、多样且可验证的结构化表格数据，通过强化学习提升长上下文推理能力。大量实验表明，表格数据显著提升多个长上下文基准上的推理表现（平均提升8.24%），甚至在跨域基准上也提升8.06%。本研究为有效后训练数据设计提供了实践指导。
原文摘要 · Abstract (English)
As real-world tasks grow increasingly complex, long-context reasoning has become a core capability for Large Language Models (LLMs). However, few studies explore which data types are effective for long-context reasoning and why. We find that structured table data with periodic structures shows strong potential for long-context reasoning. Motivated by this observation, we mathematically analyze tabular dependency structures using mutual information, revealing periodic non-vanishing dependencies in table data. Furthermore, we systematically analyze the capabilities of structured table data, conduct relevant scaling experiments, and validate its underlying mechanisms for enhancing long-context reasoning, yielding several meaningful insights. Leveraging these insights, we propose a simple yet scalable pipeline(TableLong) for synthesizing high-quality, diverse, and verifiable structured table data to boost long-context reasoning via RL. Extensive experimental results demonstrate that table data significantly enhances the long-context reasoning capability of LLMs across multiple long-context benchmarks (+8.24\% on average), and even improves performance on out-of-domain benchmarks (+8.06\% on average). We hope that our insights provide practical guidance for effective post-training data to enhance long-context reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。