让Transformer每层都能动态访问所有历史层的上下文，显著提升性能并压缩缓存。
WhiteMatter: All-to-All Cross-Layer Connections via KV Mixing
- 引入可变路由机制，每层可灵活连接所有历史层的键值对。
- 实验显示，用50%更少缓存空间仍保持50%更多层数的模型性能。
- 适合追求高效推理与高层数模型的场景，尤其关注内存优化者。
在Transformer中，每一层仅能通过自身深度产生的键值对（KV）来关注过去标记，即使解码时存在更深的表示。反馈架构虽使浅层能访问深层表示，但所有消费者层使用相同的固定连接模式。我们提出WhiteMatter，使每个注意力层都能连接到每个过去标记的所有层表示，并允许连接权重随消费者层和源标记自适应变化。对于每个标记，路由器将其L个层状态混合为k个键值通道并缓存供后续标记使用；每个消费者层关注其中一个通道。通道数k控制键值缓存大小。当k<L时，可降低缓存内存开销。在预训练实验中，WhiteMatter的表现优于层深增加50%的原始Transformer，并在键值缓存压缩50%的情况下仍保留大部分性能增益。
原文摘要 · Abstract (English)
In a Transformer, each layer attends to past tokens only through KV produced at its own depth, despite the presence of deeper representations during autoregressive decoding. Feedback architectures allow shallow consumer layers to attend to KV produced by deeper past-token representations, but give all consumer layers the same fixed connection patterns to source layers. We propose WhiteMatter, which connects every attention layer to the representations from all layers of each past token, with connection weights that can vary across consumer layers and adapt to the source token. For each token, a router implements these connections by mixing its $L$ layer states into $k$ KV channels that are cached for subsequent tokens; each consumer layer attends to one of the channels. The number of channels $k$ controls the KV-cache size. Setting $k<L$ reduces the cache's memory footprint. In our pretraining experiments, WhiteMatter outperforms a vanilla Transformer with 50% more layers and retains most of this gain with a 50% KV-cache compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。