通过重构多层潜在状态,提升大模型长文本记忆与推理一致性。
Contextual Memory Reweaving in Large Language Models Using Layered Latent State Reconstruction
- 分层重构隐藏状态,不依赖外部记忆模块强化上下文表征。
- 长序列下罕见词召回率与数值推理一致性显著提升。
- 适合需要连贯长文本生成或复杂推理的场景使用。
深度神经网络在处理长序列时面临记忆保持难题,随着序列增长,标记间依赖性减弱,导致输出连贯性与事实一致性下降。本文提出上下文记忆重织框架,通过分层潜在状态重构机制,在不引入外部记忆模块的前提下,系统性整合过往上下文嵌入,强化长序列中的标记表示。实验表明,该方法在多种序列长度下均提升了召回准确率,尤其在罕见词保留与数值推理一致性方面表现突出。计算效率分析显示,额外开销处于可接受范围,支持不同规模模型扩展。在长文本生成与模糊查询解析任务中,记忆重织显著增强输出连续性,减少不一致现象。注意力权重分布显示更结构化的分配模式,表明重织后的潜在状态提升了上下文感知能力。研究为优化语言模型的记忆机制提供了新路径,有效应对复杂多步推理任务中的长期依赖挑战。
原文摘要 · Abstract (English)
Memory retention challenges in deep neural architectures have ongoing limitations in the ability to process and recall extended contextual information. Token dependencies degrade as sequence length increases, leading to a decline in coherence and factual consistency across longer outputs. A structured approach is introduced to mitigate this issue through the reweaving of latent states captured at different processing layers, reinforcing token representations over extended sequences. The proposed Contextual Memory Reweaving framework incorporates a Layered Latent State Reconstruction mechanism to systematically integrate past contextual embeddings without introducing external memory modules. Experimental results demonstrate improvements in recall accuracy across a range of sequence lengths, with notable gains in the retention of rarely occurring tokens and numerical reasoning consistency. Further analysis of computational efficiency indicates that the additional processing overhead remains within acceptable thresholds, enabling scalability across different model sizes. Evaluations in long-form text generation and ambiguous query resolution highlight the capacity of memory reweaving to enhance continuity and reduce inconsistencies over extended outputs. Attention weight distributions reveal more structured allocation patterns, suggesting that reweaved latent states contribute to improved contextual awareness. The findings establish a framework for refining memory retention mechanisms in language models, addressing long-standing challenges in handling complex, multi-step reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。