让大模型在推理时记住关键信息，避免越算越忘。
Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoning
- 用三个可学习门控机制，在推理中持续保存和调用中间结论。
- 在GSM8K等数据集上，性能随任务难度提升反而更优，解决遗忘问题。
- 适合需要多步推理的数学与问答任务，尤其对复杂规划场景有效。
大语言模型在数学和多跳规划任务中展现出强大推理能力。CoCoNuT（连续思维链）范式通过在潜在空间中并行探索多个推理路径，超越了早期单条链的限制。然而，我们发现一个称为‘概念瓶颈’的缺陷：每轮推理中，中间隐藏状态会被覆盖，导致早期计算的关键事实随推理深度增加而丢失。实验证明，原始CoCoNuT在HotpotQA上表现不佳（10.4% EM），低于传统思维链基线（11.0% EM），且在GSM8K上随课程深度增加性能下降。为此，我们提出AGCLR（自适应门控连续潜在推理），在CoCoNuT基础上引入‘门控概念流’，通过三个可学习门控机制维持跨推理轮次的持久残差记忆：写入门用于将中间事实存入记忆，读取门用于检索相关历史状态，遗忘门用于清除无关上下文。在使用GPT-2作为基础模型的GSM8K、HotpotQA、ProsQA上评估，AGCLR在所有数据集上均实现一致提升，尤其在任务难度递增时效果更显著，有效缓解了概念瓶颈。代码已公开于https://anonymous.4open.science/r/JJJJ/README.md。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable reasoning abilities on mathematical and multi-hop planning tasks. The CoCoNuT (Chain of Continuous Thought) paradigm~\cite{hao2024coconut} extends this by enabling models to reason in latent space, exploring multiple reasoning paths simultaneously rather than committing to a single chain early on. However, we identify a limitation we term the \textbf{concept bottleneck}. At each reasoning pass, intermediate hidden states are overwritten, causing the model to lose critical facts computed in earlier steps as reasoning depth increases. We observe this empirically. On HotpotQA, vanilla CoCoNuT (10.4\% EM) fails to improve over the CoT baseline (11.0\% EM), and performance degrades with curriculum depth on GSM8K. To address this, we propose \textbf{AGCLR} (Adaptive Gated Continuous Latent Reasoning), which augments CoCoNuT with a \textit{Gated Concept Stream}. A persistent residual memory maintained across all reasoning passes, controlled by three learned gates: a \textit{write} gate that commits intermediate facts to memory, a \textit{read} gate that retrieves relevant prior states, and a \textit{forget} gate that prunes irrelevant context. Evaluated on GSM8K, HotpotQA, and ProsQA using GPT-2 as our base model, AGCLR achieves consistent improvements across all types of datasets. With the performance gap compounding as curriculum depth increases, directly resolving the concept bottleneck. Code available at https://anonymous.4open.science/r/JJJJ/README.md
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。