动态分块与选块，让大模型更好理解超长文本
Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models
- 按语义相似度动态切分长文本为不等长段落
- 用问题感知分类器选出关键段落，提升答对率
- 支持最长256k token，适配长文档问答场景
大型语言模型在处理超长文本时常面临理解困难。现有方法多采用固定长度切分，但易割裂语义连贯内容，导致歧义。为此，我们提出一种动态分块与选块方法：通过计算相邻句子的语义相似度，低相似度处自适应分割，形成可变长度段落；同时训练一个面向问题的分类器，筛选对解答关键的段落。在单跳与多跳问答基准测试中，该方法持续优于强基线模型，且在长达256k tokens的输入下仍保持稳定性能。相关数据集与代码已公开于https://github.com/ECNU-Text-Computing/DCS。
原文摘要 · Abstract (English)
Large language models (LLMs) often struggle to accurately read and comprehend extremely long texts. Current methods for improvement typically rely on splitting long contexts into fixed-length chunks. However, fixed truncation risks separating semantically relevant content, leading to ambiguity and compromising accurate understanding. To overcome this limitation, we propose a straightforward approach for dynamically separating and selecting chunks of long context, facilitating a more streamlined input for LLMs. In particular, we compute semantic similarities between adjacent sentences, using lower similarities to adaptively divide long contexts into variable-length chunks. We further train a question-aware classifier to select sensitive chunks that are critical for answering specific questions. Experimental results on both single-hop and multi-hop question-answering benchmarks show that the proposed approach consistently outperforms strong baselines. Notably, it maintains robustness across a wide range of input lengths, handling sequences of up to 256k tokens. Our datasets and code are available at the following link: https://github.com/ECNU-Text-Computing/DCS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。