让机器像大脑一样在注意力前预判重点，提速学习效率。
Beyond Attention: Toward Machines with Intrinsic Higher Mental States
- 用三元神经回路模拟思维预判，提前筛选关键信息
- 训练速度提升数个量级，计算量减少近半（如头数、层数下降）
- 适合追求高效推理的AI系统设计者，尤其视觉与语言任务
注意力机制是哺乳动物大脑与现代机器学习模型（如Transformer）的核心，但判断相关性仍是根本挑战，传统上依赖反向传播等学习算法。受新细胞神经生物学证据启发，该研究发现可通过模拟新皮层锥体细胞与不同心智状态的关联，使模型在应用注意力前就具备高层次感知与清醒思维（如想象）状态，从而预选相关信息。通过问题（Q）、线索（K）与假设（V）之间的三元神经级调节回路，实现多层次并行推理链，快速从初始偏见转向精准理解。该方法使学习速度提升数个量级，显著降低计算需求（如减少注意力头、层数与输入标记数），整体复杂度约为$/mathcal{O}(N)$，其中$N$为输入标记数。实验覆盖强化学习（如高维视觉下的CarRacing）、计算机视觉与自然语言问答任务。
原文摘要 · Abstract (English)
Attending to what is relevant is fundamental to both the mammalian brain and modern machine learning models such as Transformers. Yet, determining relevance remains a core challenge, traditionally offloaded to learning algorithms like backpropagation. Inspired by recent cellular neurobiological evidence linking neocortical pyramidal cells to distinct mental states, this work shows how models (e.g., Transformers) can emulate high-level perceptual processing and awake thought (imagination) states to pre-select relevant information before applying attention. Triadic neuronal-level modulation loops among questions ($Q$), clues (keys, $K$), and hypotheses (values, $V$) enable diverse, deep, parallel reasoning chains at the representation level and allow a rapid shift from initial biases to refined understanding. This leads to orders-of-magnitude faster learning with significantly reduced computational demand (e.g., fewer heads, layers, and tokens), at an approximate cost of $\mathcal{O}(N)$, where $N$ is the number of input tokens. Results span reinforcement learning (e.g., CarRacing in a high-dimensional visual setup), computer vision, and natural language question answering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。