发现文本嵌入模型严重偏向开头内容，影响检索准确性。
Quantifying Positional Biases in Text Embedding Models
- 通过消融实验发现开头修改比结尾修改导致相似度下降最多12.3%。
- 位置越靠后，句子重要性越低，即使忽略内容也存在此趋势。
- 适合关注检索系统鲁棒性与嵌入模型公平性的研究者阅读。
嵌入模型在信息检索和语义相似度计算中至关重要，但其对长文本及位置偏见的处理仍缺乏深入研究。本研究探讨了内容位置和输入长度对文本嵌入的影响。实验表明，无论采用何种位置编码机制，嵌入模型均显著偏好输入开头。消融实验显示，插入无关文本或删除开头内容导致的余弦相似度下降，比在结尾进行类似操作高出最多12.3%。回归分析进一步证实，随着位置远离开头，句子重要性递减，即便不依赖内容特征亦然。我们推测该现象源于预处理策略与所选位置编码方式。研究结果量化了检索系统的敏感性，为评估嵌入模型鲁棒性提供了新视角。
原文摘要 · Abstract (English)
Embedding models are crucial for tasks in Information Retrieval (IR) and semantic similarity measurement, yet their handling of longer texts and associated positional biases remains underexplored. In this study, we investigate the impact of content position and input size on text embeddings. Our experiments reveal that embedding models, irrespective of their positional encoding mechanisms, disproportionately prioritize the beginning of an input. Ablation studies demonstrate that insertion of irrelevant text or removal at the start of a document reduces cosine similarity between altered and original embeddings by up to 12.3% more than ablations at the end. Regression analysis further confirms this bias, with sentence importance declining as position moves further from the start, even with with content-agnosticity. We hypothesize that this effect arises from pre-processing strategies and chosen positional encoding techniques. These findings quantify the sensitivity of retrieval systems and suggest a new lens towards embedding model robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。