提出新型注意力机制，用单层实现长时实体追踪
Chain and Causal Attention for Efficient Entity Tracking
- 将注意力视为邻接矩阵，构建因果链式追踪结构
- 仅需1层即可完成含n次状态变化的实体追踪
- 适合需要高效长程依赖建模的追踪任务
本文研究大语言模型中变换器在实体追踪任务上的局限性。理论分析表明，变换器处理n次状态变化的实体追踪至少需要$\log_2 (n+1)$层。为解决此问题，我们提出一种高效且轻量的注意力机制改进，使其能更有效地建模长时依赖。通过将注意力视为邻接矩阵，新模型可在单层内完成实体状态追踪。实验证明，在多个实体追踪数据集上性能显著提升，同时在标准自然语言建模任务上保持竞争力。该机制不仅大幅减少所需层数，还揭示了注意力内部的结构化表示。在简单与复杂数据集上的广泛实验验证了方法的有效性。贡献包括理论洞察、改进的注意力机制和充分的实证支持。
原文摘要 · Abstract (English)
This paper investigates the limitations of transformers for entity-tracking tasks in large language models. We identify a theoretical constraint, showing that transformers require at least $\log_2 (n+1)$ layers to handle entity tracking with $n$ state changes. To address this issue, we propose an efficient and frugal enhancement to the standard attention mechanism, enabling it to manage long-term dependencies more efficiently. By considering attention as an adjacency matrix, our model can track entity states with a single layer. Empirical results demonstrate significant improvements in entity tracking datasets while keeping competitive performance on standard natural language modeling. Our modified attention allows us to achieve the same performance with drastically fewer layers. Additionally, our enhanced mechanism reveals structured internal representations of attention. Extensive experiments on both toy and complex datasets validate our approach. Our contributions include theoretical insights, an improved attention mechanism, and empirical validation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。