受果蝇脑连接组启发，用随机路由提升线性注意力的表达能力。
Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
- 对输入序列做随机重排，再用滑窗注意力处理，实现全局感知。
- 仅需 log_w n 层即可覆盖全序列，比传统方法快得多。
- 适配大模型推理与训练，提升性能且不增加计算开销。
果蝇全脑连接组包含超过13万神经元，连接概率仅为0.02%，却能实现平均4.4跳的最短路径。尽管电路结构高度有序，远距离连接却广泛分布于脑区之间，作为随机捷径实现高效全局通信。受此启发，我们提出随机注意力（Stochastic Attention, SA），作为滑窗注意力（SWA）的即插即用增强：先对标记序列进行随机重排，再进行窗口化注意力，最后恢复原始顺序。该方法在相同O(nw)每层预算下，将固定局部窗口转化为随机全局窗口。通过深度上独立采样的重排，接收域呈指数增长，仅需O(log_w n)层即可实现全序列覆盖，远优于SWA的O(n/w)。我们在两个场景验证：从零开始预训练语言模型，门控SA+SWA组合达到最佳平均零样本准确率；在Qwen3-8B和Qwen3-30B-A3B的无训练推理中，SA持续优于SWA，且在相当计算预算下可媲美或超越块混合注意力（Mixture of Block Attention）。结果表明，受连接组启发的随机路由是提升高效注意力表达力的实用范式，可与现有线性和稀疏方法互补。
原文摘要 · Abstract (English)
The whole-brain connectome of a fruit fly comprises over 130K neurons connected with a probability of merely 0.02%, yet achieves an average shortest path of only 4.4 hops. Despite being highly structured at the circuit level, the network's long-range connections are broadly distributed across brain regions, functioning as stochastic shortcuts that enable efficient global communication. Inspired by this observation, we propose Stochastic Attention (SA), a drop-in enhancement for sliding-window attention (SWA) that applies a random permutation to the token sequence before windowed attention and restores the original order afterward. This transforms the fixed local window into a stochastic global one within the same $O(nw)$ per-layer budget. Through depth, independently sampled permutations yield exponentially growing receptive fields, achieving full sequence coverage in $O(\log_w n)$ layers versus $O(n/w)$ for SWA. We validate SA in two settings: pre-training language models from scratch, where a gated SA + SWA combination achieves the best average zero-shot accuracy, and training-free inference on Qwen3-8B and Qwen3-30B-A3B, where SA consistently outperforms SWA and matches or exceeds Mixture of Block Attention at comparable compute budgets. These results suggest that connectome-inspired stochastic routing is a practical primitive for improving the expressivity of efficient attention, complementary to existing linear and sparse approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。