用监督式思维链提升大模型长文本理解能力
Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning
- 引入有监督思维链，通过合成数据引导模型逐步推理
- 在金融长文本任务中，模型准确率提升28.0%
- 适合需要可解释长文本理解的金融、法律等专业场景
大型语言模型（LLMs）已能处理长达2K至200万词元的序列，但单纯扩展输入长度并不保证有效理解。本文提出一种有监督的思维链（CoT）方法，通过合成数据增强长文本理解能力。为此，构建了金融领域合成数据集LongFinanceQA，包含从中间推理到最终结论的完整思维链，促使模型进行显式推理，提升准确性和可解释性。为生成高质量思维链，提出基于属性的代理推理框架PAI，模拟人类的属性提取、检索与总结过程。在Loong基准测试中，GPT-4o-mini搭配PAI相比标准版本提升20.0%；对LLaMA-3.1-8B-Instruct在LongFinanceQA上微调后，在金融子集上性能提升28.0%。
原文摘要 · Abstract (English)
Recent advances in Large Language Models (LLMs) have enabled them to process increasingly longer sequences, ranging from 2K to 2M tokens and even beyond. However, simply extending the input sequence length does not necessarily lead to effective long-context understanding. In this study, we integrate Chain-of-Thought (CoT) reasoning into LLMs in a supervised manner to facilitate effective long-context understanding. To achieve this, we introduce LongFinanceQA, a synthetic dataset in the financial domain designed to improve long-context reasoning. Unlike existing long-context synthetic data, LongFinanceQA includes intermediate CoT reasoning before the final conclusion, which encourages LLMs to perform explicit reasoning, improving accuracy and interpretability in long-context understanding. To generate synthetic CoT reasoning, we propose Property-based Agentic Inference (PAI), an agentic framework that simulates human-like reasoning steps, including property extraction, retrieval, and summarization. We evaluate PAI's reasoning capabilities by assessing GPT-4o-mini w/ PAI on the Loong benchmark, outperforming standard GPT-4o-mini by 20.0%. Furthermore, we fine-tune LLaMA-3.1-8B-Instruct on LongFinanceQA, achieving a 28.0% gain on Loong's financial subset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。