Llama-Nemoretriever-Colembed实现跨模态图文检索新纪录。
Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model
- 用双向注意力替换因果注意力，结合ColBERT晚交互机制。
- 3B模型在ViDoRe V1上达NDCG@5 91.0，V2上63.5，居榜首。
- 适合追求高精度跨模态检索的开发者与研究者。
为应对跨模态检索系统的日益增长需求，我们提出统一的文本-图像检索模型 llama-nemoretriever-colembed，其在多个基准测试中表现卓越。我们发布了两个版本：1B 和 3B。其中 3B 模型在 ViDoRe V1 上取得 NDCG@5 91.0，ViDoRe V2 上 63.5，截至 2025 年 6 月 27 日位列两个榜单第一。该方法基于 NVIDIA Eagle2 视觉语言模型（VLM），通过将因果注意力替换为双向注意力，并引入 ColBERT 风格的晚交互机制，在共享嵌入空间中实现细粒度多模态检索。尽管该机制显著提升检索精度，但也带来存储与效率的权衡。我们提供了对这些权衡的全面分析，并采用两阶段训练策略进一步增强模型检索能力。
原文摘要 · Abstract (English)
Motivated by the growing demand for retrieval systems that operate across modalities, we introduce llama-nemoretriever-colembed, a unified text-image retrieval model that delivers state-of-the-art performance across multiple benchmarks. We release two model variants, 1B and 3B. The 3B model achieves state of the art performance, scoring NDCG@5 91.0 on ViDoRe V1 and 63.5 on ViDoRe V2, placing first on both leaderboards as of June 27, 2025. Our approach leverages the NVIDIA Eagle2 Vision-Language model (VLM), modifies its architecture by replacing causal attention with bidirectional attention, and integrates a ColBERT-style late interaction mechanism to enable fine-grained multimodal retrieval in a shared embedding space. While this mechanism delivers superior retrieval accuracy, it introduces trade-offs in storage and efficiency. We provide a comprehensive analysis of these trade-offs. Additionally, we adopt a two-stage training strategy to enhance the model's retrieval capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。