让Transformer学会高效记忆局部模式,且可自由扩展。
Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations

- 分离记忆容量与模型宽度,用离散编号实现快速定位
- 在300亿参数模型上仍保持低内存开销,性能更优
- 记忆编号自带语义结构,适合分析模型内部表示
Transformer缺乏原生查找机制,需重复密集计算来识别和复用局部静态模式。Lngram v1通过离散潜在n-gram寻址实现与分词器无关的条件记忆,但其记忆容量受主干网络宽度限制,导致参数和激活成本高。本文提出Lngram v2,解耦路由数、记忆维度与主干宽度,并引入上下文感知的分组查询注意力读出,实现记忆容量独立扩展。零值Sink与反事实代理梯度进一步提升读出选择性与路由可训练性,同时保持硬离散寻址特性。跨多规模视觉-语言模型的实验表明,该方法持续提升性能,成功扩展至300亿参数模型。相比v1,v2显著降低总参数与激活参数,同时维持或提升语言建模性能。进一步分析显示,其离散编号保留了连续隐藏状态的显著语义结构,仅凭编号即可恢复语义,且在不同数据集间保持稳定的编号-语义关联。结果表明,Lngram v2是一种高效可扩展的潜在条件记忆机制,其离散地址还为分析模型内部表示提供了结构化接口。
原文摘要 · Abstract (English)
Transformers lack a native lookup mechanism, requiring repeated dense computation to recognize and reuse local static patterns. Lngram v1 introduces tokenizer-independent conditional memory through discrete latent n-gram addressing, but its memory capacity is coupled with the backbone width, limiting scalability due to high parameter and activation costs. We propose Lngram v2, which decouples the number of routes, memory dimension, and backbone width, and introduces a context-aware grouped-query attention readout to scale memory capacity independently. A zero-value Sink and counterfactual surrogate gradients further improve readout selectivity and routing trainability while preserving hard discrete addressing. Experiments across vision--language models (VLMs) of different scales show consistent improvements, including successful scaling to a 30B-parameter model. Compared with Lngram v1, Lngram v2 substantially reduces both total and activated memory parameters while maintaining or improving language modeling performance. Further analysis shows that its discrete IDs preserve substantial semantic structure of continuous hidden states, enabling semantic recovery from IDs alone and stable ID--semantic associations across datasets. These results establish Lngram v2 as an efficient and scalable latent conditional memory mechanism whose discrete addresses also provide a structured interface for analyzing internal model representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。