让大模型按需跳过中间层，动态调整计算路径。
Adaptive Large Language Models By Layerwise Attention Shortcuts
- 引入逐层注意力捷径，让末层可选择性关注任意中间层。
- 在语音、语言和音乐数据上均超越传统GPT架构表现。
- 注意力图显示模型能自适应地学习跨层复杂依赖关系。
Transformer架构是现代人工智能革命的基石，但其通过简单堆叠相同模块并逐层顺序处理信息。本文提出挑战这一范式，为类似大语言模型的系统引入自适应计算机制：通过注意力机制，使最终层可根据需要灵活访问所有中间层，从而实现计算上的注意力捷径。该设计使模型具备深度与上下文自适应能力。我们在语音标记、自然语言和符号音乐四个不同数据集上验证了该方法，对GPT类架构实现了更优性能。通过注意力热力图分析，我们发现模型能够学习到随输入上下文与深度自适应的复杂层间依赖关系。
原文摘要 · Abstract (English)
Transformer architectures are the backbone of the modern AI revolution. However, they are based on simply stacking the same blocks in dozens of layers and processing information sequentially from one block to another. In this paper, we propose to challenge this and introduce adaptive computations for LLM-like setups, which allow the final layer to attend to all of the intermediate layers as it deems fit through the attention mechanism, thereby introducing computational \textbf{attention shortcuts}. These shortcuts can thus make the architecture depth and context adaptive. We showcase four different datasets, namely acoustic tokens, natural language, and symbolic music, and we achieve superior performance for GPT-like architecture. We give evidence via attention maps that the models learn complex dependencies across layers that are adaptive in context and depth depending on the input tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。