用隐空间符号做词组检索，提升长文本建模效率
Lngram: N-gram Conditional Memory in Latent Space
- 在隐空间直接学习离散符号，实现无需分词的词组查找
- 长文本建模下困惑度降低12.3%，优于Transformer和Engram
- 可后置注入预训练模型，适合多模态任务快速增强
序列建模需要组合推理与局部知识检索，但标准Transformer通过密集计算处理两者。Engram部分解耦检索与主干网络，但其基于标记的键仍依赖文本分词与哈希压缩。我们提出Lngram，一种在隐空间中学习离散符号并进行N-gram查找的条件记忆模块。该设计摆脱对分词器ID的依赖，自然扩展至非文本模态。在评估设置中，Lngram优于Transformer与Engram基线，持续降低长上下文语言建模的困惑度，并可在预训练模型后置添加时有效注入领域知识。与主干网络联合训练进一步超越全微调；视觉-语言及视觉-语言-动作任务实验显示整体性能提升。LogitLens与CKA分析表明，Lngram使预测相关信息提前涌现，以有限推理与内存开销增加有效深度。代码已公开于https://github.com/zyaaa-ux/Lngram。
原文摘要 · Abstract (English)
Sequence modeling requires both compositional reasoning and local static knowledge retrieval, yet standard Transformers handle both through dense computation. Engram partially decouples retrieval from the backbone, but its token-based keys remain tied to text tokenization and hash compression. We propose Lngram, a latent-space conditional memory module that learns discrete symbols directly from hidden states and performs N-gram lookup over these symbols. This design removes the dependence on tokenizer IDs and naturally extends to non-text modalities. In our evaluated settings, Lngram outperforms Transformer and Engram baselines, consistently reduces perplexity in long-context language modeling, and effectively injects domain knowledge when added post hoc to pretrained models. Joint training with the backbone further surpasses full fine-tuning, while experiments on vision-language and vision-language-action tasks show overall gains. Analyses with LogitLens and CKA suggest that Lngram enables prediction-relevant information to emerge earlier, increasing effective depth with limited inference and memory overhead. Code is available at https://github.com/zyaaa-ux/Lngram.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。