受星形胶质细胞启发，实现高效长序列建模的新型注意力机制
RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers
- 借鉴星形胶质细胞的长期/短期可塑性，设计分段递归记忆压缩机制
- 在长程任务基准上达到媲美主流模型的准确率，内存与计算开销显著降低
- 适合追求高效长序列建模的研究者或工业部署场景
自注意力机制的二次复杂度严重制约了Transformer在长序列上的应用。本文受星形胶质细胞——生物记忆与突触调节的关键神经胶质细胞——的计算原理启发，提出一种互补于传统架构改进的高效自注意力方法。我们构建了循环记忆增强的星形形态变换器（RMAAT），集成抽象化的星形胶质细胞功能。RMAAT采用循环、分段处理策略，通过持续的记忆标记传递上下文信息。一种由模拟星形胶质细胞长期可塑性（LTP）导出的新保留因子控制的记忆压缩机制调节这些标记。段内注意力采用受星形胶质细胞短期可塑性（STP）启发的线性复杂度机制。训练使用专为循环网络设计的星形胶质细胞记忆重放反向传播（AMRB）算法。在长程任务基准（LRA）上的评估表明，RMAAT在准确率上具有竞争力，并在计算与内存效率上实现显著提升，揭示了将星形胶质细胞启发的动力学引入可扩展序列模型的潜力。
原文摘要 · Abstract (English)
The quadratic complexity of self-attention mechanism presents a significant impediment to applying Transformer models to long sequences. This work explores computational principles derived from astrocytes-glial cells critical for biological memory and synaptic modulation-as a complementary approach to conventional architectural modifications for efficient self-attention. We introduce the Recurrent Memory Augmented Astromorphic Transformer (RMAAT), an architecture integrating abstracted astrocyte functionalities. RMAAT employs a recurrent, segment-based processing strategy where persistent memory tokens propagate contextual information. An adaptive compression mechanism, governed by a novel retention factor derived from simulated astrocyte long-term plasticity (LTP), modulates these tokens. Attention within segments utilizes an efficient, linear-complexity mechanism inspired by astrocyte short-term plasticity (STP). Training is performed using Astrocytic Memory Replay Backpropagation (AMRB), a novel algorithm designed for memory efficiency in recurrent networks. Evaluations on the Long Range Arena (LRA) benchmark demonstrate RMAAT's competitive accuracy and substantial improvements in computational and memory efficiency, indicating the potential of incorporating astrocyte-inspired dynamics into scalable sequence models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。