LMN通过分层对数树结构，让长序列建模内存和计算效率大幅提升。
Logarithmic Memory Networks (LMNs): Efficient Long-Range Sequence Modeling for Resource-Constrained Environments
- 用分层对数树动态压缩历史信息，降低记忆开销。
- 注意力复杂度从O(n²)降至O(log n)，推理更快。
- 适合嵌入式设备等资源受限场景，无需位置编码。
长序列建模是自然语言处理与时间序列分析的关键任务。传统模型如循环神经网络（RNN）和变换器（Transformers）在处理长序列时存在计算与内存效率低的问题。本文提出对数记忆网络（Logarithmic Memory Networks, LMNs），采用分层对数树结构高效存储与检索历史信息。LMNs动态摘要上下文，将注意力机制的内存占用与计算复杂度从O(n²)降至O(log n)。模型使用单向量、定向注意力机制访问存储信息，并通过记忆块构建器（summarizer）层在训练时并行执行，在推理时以串行模式运行，充当内存管理系统。同时隐式编码位置信息，无需显式位置编码。这些特性使LMNs成为资源受限环境下长序列建模的高效可扩展方案。代码已开源，许可协议为MIT：https://github.com/AhmedBoin/LogarithmicMemory。
原文摘要 · Abstract (English)
Long-range sequence modeling is a crucial aspect of natural language processing and time series analysis. However, traditional models like Recurrent Neural Networks (RNNs) and Transformers suffer from computational and memory inefficiencies, especially when dealing with long sequences. This paper introduces Logarithmic Memory Networks (LMNs), a novel architecture that leverages a hierarchical logarithmic tree structure to efficiently store and retrieve past information. LMNs dynamically summarize historical context, significantly reducing the memory footprint and computational complexity of attention mechanisms from O(n2) to O(log(n)). The model employs a single-vector, targeted attention mechanism to access stored information, and the memory block construction worker (summarizer) layer operates in two modes: a parallel execution mode during training for efficient processing of hierarchical tree structures and a sequential execution mode during inference, which acts as a memory management system. It also implicitly encodes positional information, eliminating the need for explicit positional encodings. These features make LMNs a robust and scalable solution for processing long-range sequences in resource-constrained environments, offering practical improvements in efficiency and scalability. The code is publicly available under the MIT License on GitHub: https://github.com/AhmedBoin/LogarithmicMemory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。