LM重排器常被词汇相似性误导，表现不如简单匹配方法。
Language Model Re-rankers are Fooled by Lexical Similarities
- 用新提出的BM25分离度指标分析重排错误来源。
- 在DRUID数据集上LM重排器反而不如BM25基线。
- 适合关注RAG系统评估缺陷的研究者阅读。
语言模型（LM）重排器用于优化检索增强生成（RAG）中的检索结果。尽管比BM25等词法匹配方法更昂贵，但普遍认为其能更好处理语义信息及查询与答案间的关联。为检验这一假设是否成立，我们在NQ、LitQA2和DRUID三个数据集上评估了6种不同LM重排器。结果表明，LM重排器在DRUID数据集上难以超越简单的BM25基线。通过引入基于BM25得分的新分离度指标，我们揭示并定位了由词汇差异导致的重排错误。此外，我们测试了多种提升方法，发现它们主要对NQ有效。总体而言，本工作识别并解释了LM重排器的弱点，强调需要更具对抗性和现实性的数据集来评估其性能。
原文摘要 · Abstract (English)
Language model (LM) re-rankers are used to refine retrieval results for retrieval-augmented generation (RAG). They are more expensive than lexical matching methods like BM25 but assumed to better process semantic information and the relations between the query and the retrieved answers. To understand whether LM re-rankers always live up to this assumption, we evaluate 6 different LM re-rankers on the NQ, LitQA2 and DRUID datasets. Our results show that LM re-rankers struggle to outperform a simple BM25 baseline on DRUID. Leveraging a novel separation metric based on BM25 scores, we explain and identify re-ranker errors stemming from lexical dissimilarities. We also investigate different methods to improve LM re-ranker performance and find these methods mainly useful for NQ. Taken together, our work identifies and explains weaknesses of LM re-rankers and points to the need for more adversarial and realistic datasets for their evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。