为持续生成式检索设计可扩展的记忆模块,避免遗忘旧文档。
A Parametric Memory Head for Continual Generative Retrieval

- 用可调记忆头在不更新主干网络的情况下保存历史知识。
- 在MS MARCO和Natural Questions上,旧文档召回率提升30%以上。
- 适合需要持续更新的在线检索系统,如动态知识库应用。
生成式信息检索(GenIR)将检索任务整合进单一神经模型中,直接从查询解码文档标识符(docids)。这种模型即索引的架构虽简洁,但难以适应动态文档集合。与可模块化更新的系统不同,GenIR的知识以参数形式编码于权重中,标准微调方法易引发灾难性遗忘。我们发现顺序适应虽能提升新文档检索性能,却显著降低对早期文档的召回率,暴露明显的稳定-可塑性权衡。为此,提出后适应记忆调优(PAMT),一种仅通过模块化参数记忆头(PMH)进行稳定化的阶段。该方法冻结主干网络,引入固定地址的乘积键记忆体,在前缀树约束解码过程中,解码器隐藏状态稀疏查询记忆头,生成隐藏空间的残差修正;这些修正通过冻结的输出嵌入矩阵映射为分数调整,仅在树有效词上计算。此机制引导文档标识符生成,同时保持路由和主干参数不变。为减少跨片段干扰,每次会话仅更新根据解码时访问统计选择的固定预算记忆值,优先保留当前片段频繁激活、历史使用较少的条目。在MS MARCO和Natural Questions数据集上,采用连续、不相交的语料增量实验表明,PAMT显著提升对早期片段的保留能力,对新增文档性能影响极小,且每轮仅修改少量记忆值。
原文摘要 · Abstract (English)
Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries. While this model-as-index paradigm offers architectural simplicity, it is poorly suited to dynamic document collections. Unlike modular systems, where indexes are easily updated, GenIR's knowledge is parametrically encoded in its weights; consequently, standard adaptation methods such as full and parameter-efficient fine-tuning can induce catastrophic forgetting. We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices, exposing a pronounced stability-plasticity trade-off. To address this, we propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head (PMH). PAMT freezes the backbone and attaches a product-key memory with fixed addressing. During prefix-trie constrained decoding, decoder hidden states sparsely query PMH to produce residual corrections in hidden space; these corrections are mapped to score adjustments via the frozen output embedding matrix, computed only over trie-valid tokens. This guides docid generation while keeping routing and backbone parameters fixed. To limit cross-slice interference, PAMT updates only a fixed budget of memory values selected using decoding-time access statistics, prioritizing entries frequently activated by the current slice and rarely used in prior sessions. Experiments on MS MARCO and Natural Questions under sequential, disjoint corpus increments show that PAMT substantially improves retention on earlier slices with minimal impact on retrieval performance for newly added documents, while modifying only a sparse subset of memory values per session.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。