arXiv:2602.13215cs.AI2026-02被引 2

根据预测不确定性动态开启注意力,提升效率与鲁棒性

When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models

  • 用动态阈值判断是否激活注意力,仅22%的token触发
  • 在多个模型上性能优于纯递归或固定调度的混合模型
  • 适合需要高效处理长文本且对稳定性要求高的场景

循环-注意力混合模型旨在结合循环结构的高效性与注意力机制的表达力,但现有方法通常在所有位置统一使用注意力,即使递归状态已足够准确。本文提出AMOR(自适应元认知输出路由器),一种后置式混合架构,基于预测不确定性选择性调用注意力。在递归主干基础上,引入熵门控注意力模块,仅当模型输出熵超过由批次运行中位数和缩放标准差计算出的动态阈值时才激活。该机制简单、无需梯度,受不确定性驱动计算和系统1/系统2思维差异启发。在Mamba2和Gated DeltaNet主干(180M-1.5B参数)上,AMOR始终匹配或超越纯递归模型及固定调度混合基线,仅约22%的标记触发注意力。在常识推理基准和LongBench长上下文任务中表现优异,且在分布偏移下保持稳定性能。结果表明:注意力的应用时机与用量同样关键——基于预测不确定性的选择性分配能同时提升效率与鲁棒性,为统一或固定路由提供简单替代方案,并指向动态适配输入难度的自适应混合架构。

原文摘要 · Abstract (English)

Recurrent-attention hybrids aim to combine the efficiency of recurrence with the expressivity of attention, but existing approaches typically apply attention uniformly across all positions, even when the recurrent state alone is sufficient for accurate prediction. We introduce AMOR (Adaptive Metacognitive Output Router), a post-hoc hybrid architecture that selectively invokes attention based on predictive uncertainty. A recurrent backbone is augmented with entropy-gated attention blocks that activate only when the model's output entropy exceeds a dynamic threshold derived from a running batch median and scaled standard deviation. This yields a simple, gradient-free routing mechanism inspired by uncertainty-driven computation and the System 1 / System 2 distinction. Across Mamba2 and Gated DeltaNet backbones (180M-1.5B), AMOR consistently matches or outperforms both pure recurrent models and fixed-schedule hybrid baselines while invoking attention on only ~22% of tokens. It achieves strong performance on common-sense reasoning benchmarks and maintains stable long-context performance on LongBench, where prior hybrid models degrade under distribution shift. These results suggest that when attention is applied matters as much as how much: selectively allocating attention based on predictive uncertainty improves both efficiency and robustness, offering a simple alternative to uniform or fixed routing strategies and pointing toward adaptive hybrid architectures that dynamically match computation to input difficulty.

注意力机制混合模型自适应计算长文本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。