用动态MLP重新理解注意力机制，性能超越传统方法。
HyperMLP: An Integrated Perspective for Sequence Modeling
- 将自回归注意力看作上下文驱动的双层MLP，权重由历史信息生成。
- 在相同参数量下，模型在序列任务中持续优于标准softmax注意力。
- 适合研究注意力机制本质或追求高效序列建模的读者。
自注意力常被视作概率查询-键查找，促使设计保持归一化注意力分数和固定位置语义。我们提出更简洁统一的视角：自回归注意力头可视为由上下文历史实例化的动态两层MLP。在此视角下，注意力分数形成不断增长的隐藏表示，而ReLU或GLU等标准MLP激活函数自然实现输入条件下的上下文依赖记忆池选择，而非概率分布。基于此，我们提出HyperMLP和HyperGLU，通过反向偏移（滞后）布局，在特征空间与序列空间同时学习动态混合。我们提供了该结构表达能力的理论分析，并实证表明，在匹配参数预算下，HyperMLP/HyperGLU始终优于强基线的softmax注意力。
原文摘要 · Abstract (English)
Self-attention is often viewed as probabilistic query-key lookup, motivating designs that preserve normalized attention scores and fixed positional semantics. We advocate a simpler and more unified perspective: an autoregressive attention head can be viewed as a dynamic two-layer MLP whose weights are instantiated from the context history. From this view, attention scores form an ever-growing hidden representation, and standard MLP activations such as ReLU or GLU naturally implement input-conditioned selection over a context-dependent memory pool rather than a probability distribution. Based on this formulation, we introduce HyperMLP and HyperGLU, which learn dynamic mixing in both feature space and sequence space, using a reverse-offset (lag) layout to align temporal mixing with autoregressive semantics. We provide theoretical characterizations of the expressivity and implications of this structure, and empirically show that HyperMLP/HyperGLU consistently outperform strong softmax-attention baselines under matched parameter budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。