arXiv:2512.10282cs.AIcs.LG2025-12中稿 · Neural Networks被引 2

提出生物启发的连续时间注意力机制,提升时序建模精度与可解释性。

Neuronal Attention Circuit (NAC) for Representation Learning

  • 用线性微分方程建模注意力逻辑,结合线虫神经回路结构设计稀疏门控
  • 在长序列预测、自动驾驶等任务中性能优于或媲美主流连续时间模型
  • 支持神经元级可解释性,适合对模型透明度要求高的工业场景

注意力机制能提升RNN的表示学习能力,但其离散特性限制了连续时间(CT)建模。本文提出生物启发的连续时间注意力机制——神经元注意力电路(NAC),将注意力得分计算重构为带非线性耦合门的线性一阶常微分方程求解过程,其拓扑结构源自线虫(C. elegans)神经回路策略(NCPs)。NAC采用稀疏感知门替代密集投影进行查询-键映射,并引入双头稀疏主干网络,分别计算内容-目标门和可学习的时间常数门,实现高效自适应动态。为降低效率与内存开销,设计可调稀疏的亚二次Top-K成对拼接机制,仅选择关键查询-键交互。理论证明其状态稳定且近似误差有界。实验在不规则时间序列、长程预测、自动驾驶车道保持及工业寿命预测等多领域验证,NAC在准确率上达到或超越多个先进连续时间基线,同时具备神经元层面的可解释性。

原文摘要 · Abstract (English)

Attention improves representation learning over RNNs, but its discrete nature limits continuous-time (CT) modeling. We introduce Neuronal Attention Circuit (NAC), a novel, biologically inspired CT-attention mechanism that reformulates attention logit computation as the solution to a linear first-order ODE with nonlinear interlinked gates derived from repurposing the wiring of C. elegans Neuronal Circuit Policies (NCPs). NAC replaces dense projections with sparse sensory gates for query-key projections and introduces a sparse backbone network with two heads for computing content-target and learnable time-constant gates, enabling efficient adaptive dynamics. To improve efficiency and memory consumption, we implement an adaptable, sparse, subquadratic Top-K pairwise concatenation mechanism that selectively curates query-key interactions. We provide rigorous theoretical guarantees, including state stability and bounded approximation errors. Empirically, we implement NAC in diverse domains, including irregular time-series, long-range forecasting, lane-keeping for autonomous vehicles, and industrial prognostics. We observe that NAC matches or outperforms in accuracy against several CT state-of-the-art baselines, while being interpretable at the neuron cell level.

连续时间注意力机制可解释性时序建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。