arXiv:2606.12400cs.CLcs.IR2026-06

将文档拆成可组合的知识原子,提升长文档推理效率与准确率

Doc-to-Atom: Learning to Compile and Compose Memory Atoms

论文配图:Doc-to-Atom: Learning to Compile and Compose Memory Atoms
图 1 · 摘自论文原文
  • 把文档分解为带语义标签的知识原子,每个生成独立微调适配器
  • 仅加载相关知识原子,推理时内存占用降低,长文档问答准确率提升
  • 适合需要高效处理长文档的场景,如智能客服、法律分析

长输入序列是大语言模型进行文档理解与多步推理的核心,但注意力机制的二次方计算成本导致推理内存消耗大且缓慢。上下文蒸馏通过将上下文信息压缩到模型参数中缓解此问题,近期工作如 Doc-to-LoRA 将上下文蒸馏转化为单次前向传播,为每篇文档生成一个 LoRA 适配器。然而,为所有查询生成单一整体适配器会导致无关查询干扰、组合回忆能力受限,且难以扩展至长文档推理。为此,我们提出 Doc-to-Atom(Doc2Atom),一种可组合的参数化记忆框架,将每篇文档分解为具有语义类型的知识原子。每个原子编译为独立的 micro-LoRA 适配器和溯源检索键。推理时,轻量级查询路由选择并组装相关原子形成特定查询的适配器,再注入冻结的基座模型。整个系统通过多目标蒸馏框架端到端训练。在六个多样化问答基准上的实验表明,Doc2Atom 在性能上优于 Doc-to-LoRA 基线,同时降低了文档内化过程中的内存开销。

原文摘要 · Abstract (English)

Long input sequences are central to document understanding and multi-step reasoning in Large Language Models, yet the quadratic cost of attention makes inference both memory-intensive and slow. Context distillation mitigates this by compressing contextual information into model parameters, and recent work such as Doc-to-LoRA amortizes context distillation into a single forward pass that generates one LoRA adapter per document. However, producing a single monolithic adapter for all queries leads to irrelevant-query interference, limited compositional recall, and poor scalability to long-document reasoning. To address these challenges, we propose Doc-to-Atom (Doc2Atom), a compositional parametric memory framework that decomposes each document into semantically typed knowledge atoms. Each atom is compiled into an independent micro-LoRA adapter and a provenance retrieval key. At inference time, a lightweight query router selects and assembles only the relevant atoms into a query-specific adapter, which is then injected into a frozen base model. The entire system is trained end-to-end through a multi-objective distillation framework. Experiments on six diverse QA benchmarks demonstrate that Doc2Atom outperforms Doc-to-LoRA baselines while reducing the memory cost of document internalization.

参数化记忆长文档推理LoRA知识原子

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。