构建多条件跨模态检索新基准，提升商品搜索的精准匹配能力
Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
- 设计细粒度多条件检索数据集，支持跨模态组合查询
- 视觉线索提升初期排序精度，文本元数据优化长尾排序
- 适合研究多条件推理、商品检索与可解释性模型的学者
多模态大模型虽提升了跨模态检索能力，但现有评测集中于粗粒度或单条件对齐，忽略真实场景中多模态依赖约束。为此，我们提出MCMR（多条件跨模态检索）基准，涵盖上衣、下装、珠宝、鞋子和家具5个产品领域，保留丰富长文本元数据以支持组合匹配。每个查询融合互补的视觉与文本属性，要求模型同时满足所有条件才判定相关。我们评估了多种基于MLLM的检索器与视觉-语言重排序器，结果表明：(i) 不同模型存在显著模态不对称；(ii) 视觉线索主导早期排序精度，文本元数据稳定长尾排序；(iii) MLLM-based点对点重排序器通过显式验证查询-候选一致性，显著提升细粒度匹配效果。MCMR为推进组合式、约束感知、可解释的多模态理解提供了挑战性且具诊断性的评测标准。代码与数据集已开源。
原文摘要 · Abstract (English)
Recent advances in multimodal large language models (MLLMs) have substantially expanded the capabilities of multimodal retrieval, enabling systems to align and retrieve information across visual and textual modalities. Yet, existing benchmarks largely focus on coarse-grained or single-condition alignment, overlooking real-world scenarios where user queries specify multiple interdependent constraints across modalities. To bridge this gap, we introduce MCMR (Multi-Conditional Multimodal Retrieval): a large-scale benchmark designed to evaluate fine-grained, multi-condition cross-modal retrieval under natural-language queries. MCMR spans five product domains: upper and bottom clothing, jewelry, shoes, and furniture. It also preserves rich long-form metadata essential for compositional matching. Each query integrates complementary visual and textual attributes, requiring models to jointly satisfy all specified conditions for relevance. We benchmark a diverse suite of MLLM-based multimodal retrievers and vision-language rerankers to assess their condition-aware reasoning abilities. Experimental results reveal: (i) distinct modality asymmetries across models; (ii) visual cues dominate early-rank precision, while textual metadata stabilizes long-tail ordering; and (iii) MLLM-based pointwise rerankers markedly improve fine-grained matching by explicitly verifying query-candidate consistency. Overall, MCMR establishes a challenging and diagnostic benchmark for advancing multimodal retrieval toward compositional, constraint-aware, and interpretable understanding. Our code and dataset is available at https://github.com/EIT-NLP/MCMR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。