arXiv:2603.18676cs.AI2026-03

用抽象概念记忆增强注意力,实现高效且具创造力的上下文建模。

MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation

  • 引入可训练的抽象概念记忆库,模拟全局工作空间机制
  • 线性时间复杂度,比标准注意力快数倍,保持高精度
  • 适合追求高效与创造性表征的NLP、CV、语音任务

MANAR(带导航抽象概念表征的记忆增强注意力)通过实现全局工作空间理论(GWT)的原则,将标准多头注意力(MHA)泛化为上下文建模层。传统MHA虽支持全连接通信,但缺乏认知模型中假设的功能性瓶颈与全局整合机制。MANAR通过可训练的抽象概念记忆库和抽象概念表征(ACR)构建中央工作区,遵循两阶段逻辑:(i) 集成阶段,根据输入刺激检索记忆概念,汇聚形成集体“心理图像”(即ACR);(ii) 广播阶段,该全局状态导航并指导局部标记的上下文化。我们证明,实现GWT功能性瓶颈自然带来线性时间扩展,通过固定大小的ACR传递全局信息,解决了标准注意力固有的二次复杂度问题。MANAR是MHA的兼容重参数化,其投影具有相同语义角色,可通过权重复制实现预训练Transformer的知识迁移,克服了结构不兼容线性注意力替代方案的采用障碍。MANAR支持非凸上下文建模,合成的表示严格位于输入标记的凸包之外,体现GWT所描述的创造性综合。实证评估显示,它在语言(GLUE 85.1)、视觉(ImageNet-1K 83.9%)和语音(LibriSpeech WER 2.7%)任务上达到或超越强基线,是一种高效且表达力强的二次注意力替代方案。

原文摘要 · Abstract (English)

MANAR (Memory-augmented Attention with Navigational Abstract Conceptual Representation), contextualization layer generalizes standard multi-head attention (MHA) by instantiating the principles of Global Workspace Theory (GWT). While MHA enables unconstrained all-to-all communication, it lacks the functional bottleneck and global integration mechanisms hypothesized in cognitive models of consciousness. MANAR addresses this by implementing a central workspace through a trainable memory of abstract concepts and an Abstract Conceptual Representation (ACR). The architecture follows a two-stage logic that maps directly to GWT mechanics: (i) an integration phase, where retrieved memory concepts converge to form a collective "mental image" (the ACR) based on input stimuli; and (ii) a broadcasting phase, where this global state navigates and informs the contextualization of individual local tokens. We demonstrate that efficient linear-time scaling is a fundamental architectural byproduct of instantiating GWT functional bottleneck, as routing global information through a constant-sized ACR resolves the quadratic complexity inherent in standard attention. MANAR is a compatible re-parameterization of MHA with identical semantic roles for its projections, enabling knowledge transfer from pretrained transformers via weight-copy and thus overcoming the adoption barriers of structurally incompatible linear-time alternatives. MANAR enables non-convex contextualization, synthesizing representations that provably lie outside the convex hull of input tokens - a mathematical reflection of the creative synthesis described in GWT. Empirical evaluations confirm that MANAR matches or exceeds strong baselines across language (GLUE score of 85.1), vision (83.9% ImageNet-1K), and speech (2.7% WER on LibriSpeech), positioning it as an efficient and expressive alternative to quadratic attention.

注意力机制抽象表征线性注意力认知模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。