长文档嵌入存在位置与语言偏见，影响内容可发现性。
Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias
- 通过排列测试框架量化嵌入中的位置和语言偏见
- 早期段落和高资源语言（如英语）被过度表示，后期段落被边缘化
- 提出推理时注意力校准方法，提升后段落可发现性
为使文档各部分在基于嵌入的搜索中可被发现，其嵌入表示应反映每个部分。为此，我们引入一种基于排列的评估框架。结果显示，当文档较长且包含多个段落时，当前最先进的嵌入模型表现出系统性的位置与语言偏见：早期段落及高资源语言（如英语）段落被过度表示，而后期段落及低资源语言段落则被边缘化。进一步分析表明，位置偏见源于池化标记嵌入中的前向注意力分布，早期词元获得更高关注。为此，我们提出一种推理时注意力校准方法，使注意力在文档位置间更均匀分布，显著提升了后期段落的可发现性。评估框架与校准方法已在 https://github.com/impresso/fair-sentence-transformers 开源。
原文摘要 · Abstract (English)
To be discoverable in an embedding-based search process, each part of a document should be reflected in its embedding representation. To quantify any potential reflection biases, we introduce a permutation-based evaluation framework. With this, we observe that state-of-the-art embedding models exhibit systematic positional and language biases when documents are longer and consist of multiple segments. Specifically, early segments and segments in higher-resource languages like English are over-represented, while later segments and segments in lower-resource languages are marginalized. In our further analysis, we find that the positional bias stems from front-loaded attention distributions in pooling-token embeddings, where early tokens receive more attention. To mitigate this issue, we introduce an inference-time attention calibration method that redistributes attention more evenly across document positions, increasing discoverabiltiy of later segments. Our evaluation framework and attention calibration is available at https://github.com/impresso/fair-sentence-transformers
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。