所有内容路由机制都会导致表征坍缩,根源是度量不匹配。
All Routes Lead to Collapse

- 用欧氏距离和软最大值重加权,揭示注意力的本质是忽略键向量大小。
- 9个预训练模型中,不同路由方式均出现相似的表征坍缩现象。
- 即使归一化键向量,仍会坍缩,说明问题出在路由机制本身。
注意力陷阱、表征坍缩和范数分层被视为Transformer特有的病理问题。我们证明它们并非仅限于注意力机制,而是内容路由在固定相似性度量下的必然结果。我们提出重构公式:软最大值注意力本质上是基于欧氏距离的玻尔兹曼加权聚合,且键向量范数恒定,因此其得分忽略了$-\|k\ |^2$项,对键向量大小完全无感。这预测:任何度量与表示不匹配的路由器,都会通过集中路由来补偿,从而导致被路由表示的坍缩。我们在多种路由机制上验证了这一现象:在九个预训练Transformer的令牌级软最大值注意力、节点级图注意力、选择性状态空间模型、递归混合器的时间轴路由,以及深度方向上的学习残差路由中,均观察到相同特征。两个模型内消融实验表明,该现象由路由机制引起,而非偶然动态。其具体形式取决于每个路由器伴随的定位约束强度;我们调节该约束,使坍缩起始点在整个范围内移动。但机制本身具有普适性,无需范数分层——即使使用范数归一化的键向量,同样会发生集中与坍缩。我们并不声称这些模型实现了黎曼几何,而是认为几何视角是一种诊断工具,指出了平面、范数无感度量的不足。
原文摘要 · Abstract (English)
Attention sinks, representation collapse, and norm stratification are treated as transformer-specific pathologies. We show they are not specific to attention: they are what content-based routing does under a fixed similarity metric. We give a reframing identity: softmax attention is Boltzmann-weighted aggregation over Euclidean distances with constant key norms, so its score omits a $-\|k\|^2$ term and is blind to key magnitude. This predicts that any router whose metric is ill-matched to its representations should compensate, by concentrating its routing and collapsing the routed representations. We test it on routers that score and aggregate over different axes: softmax attention over tokens (nine pretrained transformers), graph attention over nodes, a selective state-space model and a recurrent mixer over time, and learned residuals over depth. All develop the same signature, and two within-model ablations show it is caused by the routing mechanism rather than by incidental dynamics. The form is contingent, set by the strength of the positional brake each router carries alongside its content score; we sweep that brake and move the onset across its whole range. The mechanism is not contingent, and it does not require norm stratification: a router with norm-normalized keys concentrates just the same. We do not claim these models implement Riemannian geometry; the geometric view is a diagnostic that names the inadequacy of the flat, norm-blind metric.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。