修复文本嵌入的均值偏差，无需训练即可提升38个模型性能。
Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB
- 提出两种无训练修正方法：直接减去均值或投影去除均值方向。
- 在MMTEB上38个模型中29个获益，平均提升达3.31，无模型性能下降。
- 均值模长越大越受益，且简单方向移除效果优于复杂降维方法。
我们发现当前句嵌入模型存在系统性偏差：每个嵌入向量可分解为偏移量与近似相同均值μ之和。研究了两种无训练修正方法——直接减去μ（R1）或投影至均值方向之外（R2），并通过一阶误差传播论证，R2能消除R1保留的均值估计误差的平行分量。在包含38个模型的海量多语言文本嵌入基准（MMTEB）上，R2带来一致分类性能提升（配对t值平均为3.31，38模型中有29个>2，无一损失）。针对五个模型的九种剂量响应消融实验显示，轻微单方向去除有益，但全主成分分析（PCA）白化会损害所有测试模型；尽管ˆμ与中心化第一主成分几何对齐弱，R2与深度为1的All-but-the-Top方法在下游任务中表现相差不超过0.18个百分点。
原文摘要 · Abstract (English)
We find that current sentence-embedding models produce outputs with a consistent bias: every embedding $e$ decomposes as $\tilde e + μ$, where the mean $μ$ is near-identical across all sentences. We study two training-free corrections -- subtracting $μ$ directly (R1), or projecting each embedding off the mean direction (R2) -- and show, via a first-order error-propagation argument, that R2 cancels the parallel component of mean-estimation error that R1 retains. Across 38 models on the Massive Multilingual Text Embedding Benchmark (MMTEB)~\citep{MMTEB}, R2 yields consistent classification gains (paired $\bar t = 3.31$, 29 of 38 models with $t>2$, zero losses), and the per-model mean norm $\Vertμ\Vert$ correlates with which models benefit most. A nine-method dose-response ablation on five models further reveals that mild single-direction removal helps, but full principal component analysis (PCA) whitening hurts every model we test, and that R2 and All-but-the-Top with depth one agree within $0.18$ pp downstream despite weak geometric alignment between $\hatμ$ and the centered top principal component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。