arXiv:2602.00015cs.CLcs.AI2026-02被引 3

用可调控记忆库提升大模型长程推理能力,解决信息丢失问题。

G-MemLLM: Gated Latent Memory Augmentation for Long-Context Reasoning in Large Language Models

  • 引入可训练的潜在记忆库,通过门控机制选择性更新记忆
  • 在ZsRE上使Llama 3.1-8B准确率提升13.3%,HotpotQA F1提高6.89点
  • 适用于需要多跳推理的问答与关系抽取任务

大语言模型在自然语言理解方面表现出色,但受限于上下文窗口容量及多跳推理中长期事实一致性维护的困难。现有方法如上下文压缩或循环标记虽有应用,却常出现“上下文腐化”或信息随时间稀释的问题。本文提出G-MemLLM,一种将冻结的LLM主干与可训练的潜在记忆库结合的记忆增强架构。其核心创新是采用类似GRU的门控更新逻辑,实现对记忆槽的选择性更新、保留或覆盖,避免递归系统中常见的知识梯度消失。我们在HotpotQA和零样本关系抽取(ZsRE)基准上评估了从GPT-2(124M)到Llama 3.1(8B)的多个规模模型。结果表明,G-MemLLM显著提升多跳推理与关系精度:在ZsRE上,Llama 3.1-8B准确率提升13.3%;在HotpotQA上,GPT-2的Answer F1提升8.56点,Llama 3.1-8B的Supporting Fact F1提升6.89点。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding, yet they remain constrained by the finite capacity of their context windows and the inherent difficulty of maintaining long-term factual consistency during multi-hop reasoning. While existing methods utilize context compression or recurrent tokens, they often suffer from ``context rot'' or the dilution of information over long horizons. In this paper, we propose \textbf{G-MemLLM}, a memory-augmented architecture that integrates a frozen LLM backbone with a trainable \textbf{Latent Memory Bank}. Our key innovation is a GRU-style gated update logic that allows the model to selectively update, preserve, or overwrite latent memory slots, preventing the vanishing gradients of knowledge common in recurrent systems. We evaluate G-MemLLM across scales, from GPT-2 (124M) to Llama 3.1 (8B), on the HotpotQA and Zero-Shot Relation Extraction (ZsRE) benchmarks. Our results demonstrate that G-MemLLM significantly enhances multi-hop reasoning and relational precision, achieving a 13.3\% accuracy boost on ZsRE for Llama 3.1-8B, and it also yields improvements across model scales, boosting Answer F1 by 8.56 points for GPT-2 and increasing Supporting Fact F1 by 6.89 points for Llama 3.1-8B on HotpotQA.

长程推理记忆机制大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。