提出三隐状态压缩模型，高效捕捉长程词元关联。
Generic Triple-Latent Compression with Gated Associative Retrieval
- 引入三隐状态结构，融合运行词元与压缩记忆路径
- 在WikiText-2和MiniMind上超越小规模Transformer基线
- 门控检索提升关联召回率，适合需要记忆推理的场景
我们研究通用三隐状态序列模型，该模型通过维护运行词元状态和压缩的成对记忆路径，无需针对特定基准进行解析即可捕捉高阶词元交互。三隐状态家族在字节级WikiText-2和基于分词器的MiniMind语言建模基准上均优于小型Transformer基线。进一步采用侧重回忆的门控键值检索扩展后，关联回忆能力得到提升，但当前参考实现仍存在种子敏感且速度较慢的问题。
原文摘要 · Abstract (English)
We study generic triple-latent sequence models that maintain a running token state and compressed pair-memory pathway to capture higher-order token interactions without benchmark-specific parsing. The triple-latent family improves a small Transformer baseline on byte-level WikiText-2 and on a tokenizer-based MiniMind language-model benchmark, while a recall-focused gated key-value retrieval extension improves associative recall but remains seed-sensitive and much slower in the current reference implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。