arXiv:2604.19816cs.AI2026-04被引 1

通过动态时间注意力机制,让模型调控复杂系统中的协同现象。

Emergence Transformer: Dynamical Temporal Attention Matters

论文配图:Emergence Transformer: Dynamical Temporal Attention Matters
图 1 · 摘自论文原文
  • 引入随时间变化的查询、键、值矩阵,实现动态时间注意力
  • 邻居型注意力始终增强振荡同步,自注意力在特定权重下最优
  • 适用于社交共识调控与神经网络持续学习,避免灾难性遗忘

Transformer架构的成功源于注意力机制对序列数据长程相互作用的利用,从而在大语言模型与数据分布间产生涌现一致性。然而,时间注意力——即时间序列中不同形式的长程相互作用——在包括量子、生物物理或气候系统在内的复杂系统涌现现象中尚未被充分探索。本文设计了随时间变化的查询、键和值矩阵的动态时间注意力(DTA),提出一种涌现Transformer。该架构使每个组件能通过动态注意力核与其自身或邻近状态交互,从而促进或抑制组件间的涌现一致性。有趣的是,邻近型DTA始终促进振荡一致性,而自注意力在特定权重下达到最优,因其对网络结构具有非单调依赖性。实际应用中,我们展示了DTA如何重塑社会一致性,提供增强共识或保持多元性的策略。进一步将DTA应用于典型霍普菲尔德神经网络,实现了无灾难性遗忘的涌现持续学习。这些结果为仅通过DTA调控网络动力学中的涌现现象奠定了基础并提供了直接范式。

原文摘要 · Abstract (English)

The Transformer, a breakthrough architecture in artificial intelligence, owes its success to the attention mechanism, which utilizes long-range interactions in sequential data, enabling the emergent coherence between large language models (LLMs) and data distributions. However, temporal attention, that is, different forms of long-range interactions in temporal sequences, has rarely been explored in emergence phenomenon of complex systems including oscillatory coherence in quantum, biophysical, or climate systems. Here, by designing dynamical temporal attention (DTA) with time-varying query, key, and value matrices, we propose an Emergence Transformer. This architecture allows each component to interact with its own or its neighbors' past states through dynamical attention kernels, thereby enabling the promotion and/or suppression of the emergent coherence of components. Interestingly, we uncover that neighbor-DTA consistently promotes oscillatory coherence, whereas self-DTA exhibits an optimal attention weight for coherence enhancement, owing to its non-monotonic dependence on network structure. Practically, we demonstrate how DTA reshapes social coherence, suggesting strategies to either enhance agreement or preserve plurality. We further apply DTA to the paradigmatic Hopfield neural network, achieving emergent continual learning without catastrophic forgetting. Together, these results lay a foundation and provide an immediate paradigm for modulating emergence phenomenon in networked dynamics only using DTA.

注意力机制涌现现象动态建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。