提出高阶注意力机制，提升模型对复杂关系的捕捉能力。
How Smoothing is N-simplicial Attention?
- 用N-单纯形注意力建模多令牌高阶交互关系
- 引入轻量级单纯形选择策略，降低计算开销
- 揭示注意力平滑导致过平滑问题，为优化提供方向
从纯MLP到可学习的图消息传递机制已成为当前先进模型的基础，尽管存在计算代价（如GAT或Transformer）。本文进一步提出N-单纯形注意力，将传统的成对令牌相似性扩展至更高阶交互，并适配旋转位置编码（RoPE）。为应对复杂度上升，提出一种低成本的单纯形选择机制，使模型能聚焦于任务敏感的交互。此外，通过推导利普希茨上界并实证表明，即使引入高阶交互，注意力本身仍会因平滑而出现过平滑问题。
原文摘要 · Abstract (English)
Going from pure Multilayer Perceptron (MLP) to a learnable graph message-passing mechanism at each layer has been foundational to state-of-the-art results, despite the computational trade-off (e.g. GATs or Transformers). To go a step further, in this work, we introduce N-simplicial attention, going from pairwise token similarity to higher-order interactions, and adapt it for Rotary Position Embeddings (RoPE). To help manage the increased complexity, we propose a cost-effective simplex selection enabling the model to focus its computation load onto the more task-sensitive interactions. Beyond these core mechanisms, we study how smoothing N-simplicial attention is by deriving a Lipschitz upper-bound and by demonstrating that by itself it also suffers from over-smoothing, despite opening the attention message-passing to higher-order interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。