让Transformer自己决定何时计算,提升效率。
Subjective Depth and Timescale Transformers: Learning Where and When to Compute
- 基于贝叶斯意外信号动态分配计算资源
- 减少75%自注意力计算,降低50%缓存需求
- 适合追求高效推理的长序列模型研究者
标准Transformer架构中固定的计算分配限制了其效率与可扩展性,尤其在大规模模型和长序列场景下。为此,我们提出主观深度变换器(SDT)和主观时标变换器(STT),通过贝叶斯意外信号动态路由计算,学习在解码器仅架构中何处、何时进行计算。SDT在解码器堆栈中交替引入决策层与动态层:决策层计算完整块的后验与轻量先验,动态层则基于贝叶斯意外(预期与非预期变化)采用固定容量的Top-K路由,保持静态计算图。STT将这种条件计算扩展至时间维度:一个过渡网络预测残差更新,形成时间上的“变化假设”,由路由器决定每个令牌是否动态执行或跳过变换器块,从而管理键值缓存贡献。两者均表现出训练过程中从新颖性到预测驱动的门控转变,表明与基于意外的原则一致。尽管计算容量降低,仍提供了关于条件计算中计算-精度权衡的初步洞察。所提架构建立了一个灵活的高效框架,在每个计算跳过层中实现75%的自注意力计算减少和50%的键值缓存需求降低,为更高效的模型发展提供路径。
原文摘要 · Abstract (English)
The rigid, uniform allocation of computation in standard Transformer (TF) architectures can limit their efficiency and scalability, particularly for large-scale models and long sequences. Addressing this, we introduce Subjective Depth Transformers (SDT) and Subjective Timescale Transformers (STT), two distinct architectures that leverage Bayesian surprise signals to dynamically route computation, learning where and when to compute within decoder-only TFs. SDT augments a decoder-only stack with alternating Decision and Dynamic layers: a Decision layer computes a full block 'posterior' and a lightweight 'prior,' while a Dynamic layer employs fixed-capacity Top-K routing based on Bayesian surprise (Expected and Unexpected Change), maintaining a static compute graph. STT extends this conditional computation to the temporal domain: a transition network predicts residual updates, forming a temporal 'change hypothesis' that informs a router to dynamically execute or bypass TF blocks for each token, managing KV-cache contributions. Both architectures exhibit the predicted shift from novelty to prediction driven gating over training, suggesting alignment with surprise based principles. While operating at reduced capacity, they offer preliminary insights into the compute-accuracy trade-offs of conditional computation. The proposed architectures establish a flexible framework for efficiency, reducing self-attention computation by 75% and KV-cache requirements by 50% within each compute skipping layer, setting a pathway for more efficient models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。