解决注意力残差模型中的异常值与注意力黑洞问题，提升推理稳定性和量化鲁棒性。
Attention Sinks and Outliers in Attention Residuals
- 通过层间零信号设计软softmax1零空间，耦合令牌级零证据实现深度路由优化。
- 在多数据集上降低最大无穷范数9.26%、平均峰度2.60%，量化后困惑度下降75.85%。
- 适合关注模型量化与推理稳定性、尤其是使用AttnResidual结构的研究者。
我们提出OASIS，一种基于层间零信号的异常值与注意力黑洞感知技术。由于注意力残差（AttnResidual）架构引入额外的深度方向归一化通道，虽提升了层间路由灵活性，但也加剧了注意力黑洞、激活异常值问题，导致推理稳定性与量化鲁棒性下降。OASIS通过构建基于Softmax1的零空间，并将令牌级零证据与层间零信号耦合，实现深度路由优化，减少由黑洞主导的路由路径，增强结构鲁棒性。理论上，我们证明了AttnResidual的双重归一化设计会强化黑洞形成与量化脆弱性。实验中，在三个真实世界数据集上对比五种基线，结果一致显示性能提升：在评估设置下，最大无穷范数平均降低9.26%，平均峰度降低2.60%；在W8A8量化下困惑度下降75.85%；在W4A4下GSM8K Pass@1提升12.42%。
原文摘要 · Abstract (English)
We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel, they improve inter-layer routing flexibility but also exacerbate attention sinks, activation outliers, and the resulting degradation in inference stability and quantization robustness. OASIS addresses this issue by introducing a Softmax1-based null space and coupling token-level null evidence to depth routing through an inter-layer null signal, thereby reducing sink-dominated routing and improving structural robustness. Theoretically, we show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness. Experimentally, we compare OASIS against five baselines on three real-world datasets and observe consistent improvements in both attention sink and post-quantization performance. Notably, OASIS achieves an average reduction of 9.26% in maximum infinity norm and 2.60% in average kurtosis across the evaluated settings, while lowering perplexity by 75.85% under W8A8 and improving GSM8K Pass@1 by 12.42% under W4A4.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。