发现主流相似度度量受序列长度干扰,提出更可靠的替代方案
Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative

- 揭示均值池化余弦相似度随序列变长而单调上升的缺陷
- 实验证明长度解释了52%~75%的跨语言相似性差异
- 推荐使用对长度不敏感的CKA作为默认比较指标
均值池化余弦相似度是跨语言、模态和任务比较神经表示的默认度量。我们证明该度量并非长度不变:在现代Transformer表示的各向异性下,其值随序列长度单调增长,与内容无关。在HumanEvalPack上,四个代码LLM中,长度比率单独解释了0.52至0.75的跨语言'Python接近度'方差;而AST深度和共享词元比例带来的额外解释方差不足3%。替换为中心核对齐(CKA)后,解释方差减少83%,且长度系数符号反转(β_len: +0.86 → -0.37)。在Mistral-7B的平行WMT语料对上同样观察到类似模式(英语-法语R²=0.23,英语-德语R²=0.33,CKA均<0.01)。在CLIP ViT-B/32中,均值池化相比结尾符池化显著降低长度效应(R²: 0.21 → <0.01),与理论预测一致。因此建议以长度不变的度量如CKA作为默认,此前基于均值池化余弦得出的跨语言表征收敛结论需重新审视。
原文摘要 · Abstract (English)
Mean-pooled cosine similarity is the default metric for comparing neural representations across languages, modalities, and tasks. We establish that this metric is not length-invariant: under the anisotropy that characterizes modern transformer representations, mean-pooled cosine grows monotonically in sequence length, independent of representational content. Empirically, on HumanEvalPack across four code LLMs, the length ratio alone explains $R^2 = 0.52$--$0.75$ of cross-language "Python proximity," while AST depth and shared-token fraction add less than 3% of explained variance beyond length. Substituting Centered Kernel Alignment (CKA) reduces explained variance by 83% and reverses the sign of the length coefficient ($β_{\mathrm{len}}: +0.86 \to -0.37$). The same pattern holds in Mistral-7B on parallel WMT pairs ($R^2 = 0.23$ EN-FR, $R^2 = 0.33$ EN-DE for cosine; $R^2 < 0.01$ for CKA). In CLIP ViT-B/32, mean-pooling reduces the length effect relative to EOS-pooling ($R^2: 0.21 \to {<}0.01$), as predicted by the theory's dependence on anisotropy. We argue that length-invariant metrics such as CKA should be the default for cross-representation comparisons, and that recent claims of cross-lingual representational convergence built on mean-pooled cosine warrant re-examination.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。