Gecko模型高效处理任意长度序列，无需外接技术即可稳定支持400万词长上下文。
Gecko: An Efficient Neural Architecture Inherently Processing Sequences with Arbitrary Lengths
- 基于门控注意力与指数移动平均，引入时间步衰减归一化等新组件。
- 70亿参数下训练损失1.68，优于Llama2-7B和Megalodon-7B，接近Llama2-13B。
- 原生支持超长序列，可从4倍于注意力窗口的上下文中准确检索信息。
设计一种能高效且原生处理任意长度序列的统一神经网络是序列建模中的核心挑战。Transformer的二次复杂度和弱长序列外推能力限制了其在长序列上的扩展性。本文提出Gecko，继承Mega和Megalodon（带门控注意力的指数移动平均）的设计，并引入时间步衰减归一化、滑动块注意力机制和自适应工作记忆等多个技术组件，以增强对长程依赖的捕捉能力。在70亿参数规模和2万亿训练标记的受控预训练对比中，Gecko在效率和长上下文可扩展性上表现更优。其训练损失为1.68，显著优于Llama2-7B（1.75）和Megalodon-7B（1.70），接近Llama2-13B（1.67）。值得注意的是，不依赖任何上下文扩展技术，Gecko具备原生长上下文处理与信息检索能力，可稳定处理长达400万词的序列，并从比其注意力窗口长4倍的上下文中准确提取信息。代码已开源。
原文摘要 · Abstract (English)
Designing a unified neural network to efficiently and inherently process sequential data with arbitrary lengths is a central and challenging problem in sequence modeling. The design choices in Transformer, including quadratic complexity and weak length extrapolation, have limited their ability to scale to long sequences. In this work, we propose Gecko, a neural architecture that inherits the design of Mega and Megalodon (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability to capture long range dependencies, including timestep decay normalization, sliding chunk attention mechanism, and adaptive working memory. In a controlled pretraining comparison with Llama2 and Megalodon in the scale of 7 billion parameters and 2 trillion training tokens, Gecko achieves better efficiency and long-context scalability. Gecko reaches a training loss of 1.68, significantly outperforming Llama2-7B (1.75) and Megalodon-7B (1.70), and landing close to Llama2-13B (1.67). Notably, without relying on any context-extension techniques, Gecko exhibits inherent long-context processing and retrieval capabilities, stably handling sequences of up to 4 million tokens and retrieving information from contexts up to $4\times$ longer than its attention window. Code: https://github.com/XuezheMax/gecko-llm
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。