arXiv:2512.03514cs.IRcs.AI2025-12被引 1

M3DR实现跨语言图文文档检索,支持22种语言且性能提升1.5倍

M3DR: Towards Universal Multilingual Multimodal Document Retrieval

  • 利用合成多语言文档数据,通过对比学习构建统一跨模态表示
  • 在22种语言上实现稳定表现,跨语言检索性能提升约150%
  • 适配不同模型架构,适用于真实多语言场景的检索系统开发

多模态文档检索系统在对齐视觉与文本内容方面取得显著进展,但多数方法仍以英语为中心,限制了其在多语言环境下的应用。本文提出M3DR(Multilingual Multimodal Document Retrieval)框架,旨在跨越语言鸿沟,支持多样语言和文化背景下的应用。M3DR利用合成多语言文档数据,可泛化至不同视觉-语言架构和模型规模,实现稳健的跨语言、跨模态对齐。通过对比学习,模型学习到统一的文本与文档图像表示,具备良好跨语言迁移能力。我们在22种语言类型差异较大的语言上验证该能力,展示出一致的表现和适应性。我们还构建了一个涵盖真实多语言场景的综合基准,评估模型在单语、多语及混合语言设置下的表现。M3DR同时适配单一密集向量与ColBERT风格的分词级多向量检索范式。所提出的NetraEmbed与ColNetraEmbed模型在跨语言检索任务中达到最先进水平,相对性能提升约150%。

原文摘要 · Abstract (English)

Multimodal document retrieval systems have shown strong progress in aligning visual and textual content for semantic search. However, most existing approaches remain heavily English-centric, limiting their effectiveness in multilingual contexts. In this work, we present M3DR (Multilingual Multimodal Document Retrieval), a framework designed to bridge this gap across languages, enabling applicability across diverse linguistic and cultural contexts. M3DR leverages synthetic multilingual document data and generalizes across different vision-language architectures and model sizes, enabling robust cross-lingual and cross-modal alignment. Using contrastive training, our models learn unified representations for text and document images that transfer effectively across languages. We validate this capability on 22 typologically diverse languages, demonstrating consistent performance and adaptability across linguistic and script variations. We further introduce a comprehensive benchmark that captures real-world multilingual scenarios, evaluating models under monolingual, multilingual, and mixed-language settings. M3DR generalizes across both single dense vector and ColBERT-style token-level multi-vector retrieval paradigms. Our models, NetraEmbed and ColNetraEmbed achieve state-of-the-art performance with ~150% relative improvements on cross-lingual retrieval.

多语言图文检索跨模态对比学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。