arXiv:2606.18801cs.IRcs.AI2026-06

通过索引阶段的语义调和,解决多语言检索中的语言偏见问题。

SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval

论文配图:SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval
图 1 · 摘自论文原文
  • 利用双语对照对估计目标语言相对偏移向量
  • 在索引时减去语言偏移,平衡不同语言文档排序
  • 无需训练,兼容多种模型,适合多语言系统部署

随着大规模多语言语料库的快速扩展,多语言信息检索(MLIR)已成为全球信息获取的关键技术。MLIR使用户能够用单语查询从多语言文本集合中检索语义相关文档。然而,现有多语言密集检索模型常表现出对与查询同语种文档的强烈偏好,导致严重语言偏见:即使其他语言文档包含更相关的语义信息,排名靠前的结果仍被特定语言主导。为此,我们提出SHIFT,一种可在索引阶段应用的无训练方法。具体而言,SHIFT利用平行翻译对估计目标语言相对于源语言的相对语言向量,并在索引时将该向量从文档嵌入中减去,以校正语言特异性偏移。在四个MLIR基准和多种密集检索模型上的综合评估表明,SHIFT能有效缓解语言偏见并提升MLIR性能。

原文摘要 · Abstract (English)

With the rapid expansion of massive multilingual corpora, Multilingual Information Retrieval (MLIR) has emerged as a critical technology for global information access. MLIR enables users to retrieve semantically relevant documents from multilingual text collections using a single-language query. However, recent multilingual dense retrieval models often exhibit a strong preference for documents in the same language as the query. This leads to severe language bias, where top-ranked results are dominated by documents of specific languages, even when documents in other languages contain more semantically relevant information. To address this issue, we propose SHIFT, a training-free method applicable in the indexing stage. Specifically, SHIFT utilizes parallel translation pairs to estimate a relative language vector for each target language with respect to a source language. Subsequently, SHIFT corrects the language-specific offset by subtracting this relative language vector from document embeddings during indexing. Our comprehensive evaluation across four MLIR benchmarks and diverse dense retrieval models confirms that SHIFT can effectively mitigate language bias and enhance MLIR performance.

多语言检索语言偏见语义调和索引优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。