构建多条件检索基准，揭示现有系统在复杂查询下的性能短板
MultiConIR: Towards multi-condition Information Retrieval
- 设计跨五领域的多条件检索评测基准，测试三类核心能力
- 15个模型均随查询复杂度上升表现骤降，尤其难保持相关性单调性
- 发现条件位置与查询风格显著影响检索效果，提示大模型优势差距
多条件信息检索（MultiConIR）是现有系统尚未充分探索的重要挑战。本文提出一个专门针对五个不同领域中细微多条件查询场景的评估基准，系统性地通过三项关键任务——复杂度鲁棒性、相关性单调性与查询格式敏感性——评估检索与重排序模型的能力。对15个模型的广泛实验显示，大多数检索器和重排序器在查询复杂度增加时出现严重性能下降。主要缺陷包括普遍无法维持相关性单调性，以及对查询风格和条件位置高度敏感。GPT-4o的优异表现揭示了传统信息检索系统与先进大语言模型在处理复杂自然语言查询上的显著差距。此外，本研究深入分析了重排序器性能退化的成因，并考察了查询中条件位置对相似性判断的影响，为推动信息检索系统应对复杂搜索场景提供了关键洞见。代码与数据集已公开于 https://github.com/EIT-NLP/MultiConIR
原文摘要 · Abstract (English)
Multi-condition information retrieval (IR) presents a significant, yet underexplored challenge for existing systems. This paper introduces MultiConIR, a benchmark specifically designed to evaluate retrieval and reranking models under nuanced multi-condition query scenarios across five diverse domains. We systematically assess model capabilities through three critical tasks: complexity robustness, relevance monotonicity, and query format sensitivity. Our extensive experiments on 15 models reveal a critical vulnerability: most retrievers and rerankers exhibit severe performance degradation as query complexity increases. Key deficiencies include widespread failure to maintain relevance monotonicity, and high sensitivity to query style and condition placement. The superior performance of GPT-4o reveals the performance gap between IR systems and advanced LLM for handling sophisticated natural language queries. Furthermore, this work delves into the factors contributing to reranker performance deterioration and examines how condition positioning within queries affects similarity assessment, providing crucial insights for advancing IR systems towards complex search scenarios. The code and datasets are available at https://github.com/EIT-NLP/MultiConIR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。