改进Transformer注意力，让内容信息绕过位置感知模块提升效果。
Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V
- 在归一化后引入非线性预投影，构建与位置无关的丰富特征。
- 通过内容跳连使关键信息绕过位置注意力，160M模型上准确率提升40.6%。
- 深层网络更依赖内容跳连，适合追求性能优化的NLP研究者。
我们提出两种互补的Transformer注意力块改进方法。首先，在层归一化与查询/键/值投影之间插入非线性预投影MLP，于位置编码前以无位置方式构建更丰富的特征。其次，引入内容跳连机制，将预投影特征绕过注意力模块，使内容信息在有利时直接传递。在Pythia-160M和410M模型上的冻结探测实验中，联合方法表现最优：160M规模下LAMBADA准确率提升40.6%，困惑度降低39%。学习到的跳连权重显示稳定模式：深层网络比浅层更强烈启用内容跳连，表明深层更受益于不经过位置注意力的内容信息。所有修改均不增加键/值缓存开销。
原文摘要 · Abstract (English)
We propose two complementary modifications to transformer attention blocks. First, a non-linear pre-projection MLP is inserted between layer norm and Q/K/V projections, constructing richer features in a position-agnostic manner before any positional encoding is applied. Second, a content skip connection routes the pre-projection's features around the attention mechanism, allowing content information to bypass position-aware attention where beneficial. In frozen-probe experiments on Pythia-160M and 410M, the combined approach achieves the strongest results across methods: +40.6% LAMBADA accuracy and -39% perplexity at 160M scale. Learned skip connection weights reveal a consistent pattern across model sizes: later transformer layers activate the content bypass more strongly than earlier layers, suggesting that deeper layers benefit from content information that does not pass through positional attention. All modifications add no K/V cache overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。