将知识与推理分离，用隐式令牌提升长文本推理效率
Decoupled Reasoning with Implicit Fact Tokens (DRIFT): A Dual-Model Framework for Efficient Long-Context Inference
- 用轻量知识模型生成查询相关的隐式事实令牌
- 在相同模型规模下，长文本任务表现优于主流基线
- 适合需要高效处理长文档的场景，如法律、医疗分析
将动态知识融入大语言模型仍面临事实数据与推理模式纠缠的挑战。现有方法如非参数检索增强生成（RAG）或参数化知识编辑，常受限于有限上下文窗口、检索噪声或灾难性遗忘。本文提出DRIFT，一种新型双模型架构，显式分离知识提取与推理过程。不同于静态提示压缩，DRIFT通过轻量级知识模型将文档片段动态压缩为条件查询的隐式事实令牌，将其投影至推理模型嵌入空间，替代冗余原文，在保持推理准确率的同时显著减少输入长度。大量实验表明，DRIFT在长上下文任务中显著优于同类规模模型的强基线。该方法为扩展大模型有效上下文窗口与推理能力提供了可扩展、高效的范式。代码已开源：https://github.com/Lancelot-Xie/DRIFT。
原文摘要 · Abstract (English)
The integration of extensive, dynamic knowledge into Large Language Models (LLMs) remains a significant challenge due to the inherent entanglement of factual data and reasoning patterns. Existing solutions, ranging from non-parametric Retrieval-Augmented Generation (RAG) to parametric knowledge editing, are often constrained in practice by finite context windows, retriever noise, or the risk of catastrophic forgetting. In this paper, we propose DRIFT, a novel dual-model architecture designed to explicitly decouple knowledge extraction from the reasoning process. Unlike static prompt compression, DRIFT employs a lightweight knowledge model to dynamically compress document chunks into implicit fact tokens conditioned on the query. These dense representations are projected into the reasoning model's embedding space, replacing raw, redundant text while maintaining inference accuracy. Extensive experiments show that DRIFT significantly improves performance on long-context tasks, outperforming strong baselines among comparably sized models. Our approach provides a scalable and efficient paradigm for extending the effective context window and reasoning capabilities of LLMs. Our code is available at https://github.com/Lancelot-Xie/DRIFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。