提出新模型解决长文本理解中信息遗漏与注意力分散问题
FltLM: An Intergrated Long-Context Large Language Model for Effective Context Filtering and Understanding
- 用动态软掩码机制过滤无关内容，聚焦关键信息
- 单次前向传播即可完成长文档问答，效率更高
- 在复杂问答任务中超越现有方法，适合长文本应用
长上下文大语言模型（LLMs）显著推进了自然语言处理，使跨长文档和多语料库的文本处理成为可能。然而，仍面临两大挑战：‘中间信息丢失’现象导致关键中间内容被忽略，以及过长上下文引发的注意力分散问题。为此，我们提出上下文过滤语言模型（FltLM），一种新型集成式长上下文模型，可有效提升多文档问答任务的表现。FltLM创新性地引入带有软掩码机制的上下文过滤器，动态识别并排除无关内容，使模型更专注地理解和推理相关信息。该方法不仅缓解上述两个问题，还支持单次前向传播完成推理。实验表明，FltLM在复杂问答场景中显著优于监督微调和基于检索的方法，为更准确可靠的长上下文自然语言理解提供了可行方案。
原文摘要 · Abstract (English)
The development of Long-Context Large Language Models (LLMs) has markedly advanced natural language processing by facilitating the process of textual data across long documents and multiple corpora. However, Long-Context LLMs still face two critical challenges: The lost in the middle phenomenon, where crucial middle-context information is likely to be missed, and the distraction issue that the models lose focus due to overly extended contexts. To address these challenges, we propose the Context Filtering Language Model (FltLM), a novel integrated Long-Context LLM which enhances the ability of the model on multi-document question-answering (QA) tasks. Specifically, FltLM innovatively incorporates a context filter with a soft mask mechanism, identifying and dynamically excluding irrelevant content to concentrate on pertinent information for better comprehension and reasoning. Our approach not only mitigates these two challenges, but also enables the model to operate conveniently in a single forward pass. Experimental results demonstrate that FltLM significantly outperforms supervised fine-tuning and retrieval-based methods in complex QA scenarios, suggesting a promising solution for more accurate and reliable long-context natural language understanding applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。