RetNet用线性复杂度替代Transformer，提升长序列处理效率。
A Survey of Retentive Network
- 用保留机制融合循环与注意力，实现线性推理时间。
- 在文本、语音、时序任务中表现稳定，跨领域效果强。
- 适合需要高效长序列建模的研究者与工程应用
Retentive Network（RetNet）是神经网络架构的重要进展，为Transformer提供了高效的替代方案。Transformer依赖自注意力建模依赖关系，但因二次复杂度导致长序列处理时内存开销大、可扩展性差。RetNet通过引入保留机制，统一了循环的归纳偏置与注意力的全局依赖建模能力，实现线性时间推理，支持高效长上下文建模，并兼容全并行训练。该架构因其跨领域的持续有效性受到广泛关注，在自然语言处理、语音识别和时序分析等任务中均表现出色。然而，现有文献尚缺乏对RetNet的系统综述。本文首次全面梳理RetNet的架构设计、核心创新及其多样化应用，探讨其面临的主要挑战，并提出未来研究方向，以推动其在学术与实际部署中的进一步发展。
原文摘要 · Abstract (English)
Retentive Network (RetNet) represents a significant advancement in neural network architecture, offering an efficient alternative to the Transformer. While Transformers rely on self-attention to model dependencies, they suffer from high memory costs and limited scalability when handling long sequences due to their quadratic complexity. To mitigate these limitations, RetNet introduces a retention mechanism that unifies the inductive bias of recurrence with the global dependency modeling of attention. This mechanism enables linear-time inference, facilitates efficient modeling of extended contexts, and remains compatible with fully parallelizable training pipelines. RetNet has garnered significant research interest due to its consistently demonstrated cross-domain effectiveness, achieving robust performance across machine learning paradigms including natural language processing, speech recognition, and time-series analysis. However, a comprehensive review of RetNet is still missing from the current literature. This paper aims to fill that gap by offering the first detailed survey of the RetNet architecture, its key innovations, and its diverse applications. We also explore the main challenges associated with RetNet and propose future research directions to support its continued advancement in both academic research and practical deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。