轻量多模态编码器NeoMME，高效处理图文混合检索任务。
NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference
- 单塔双向Transformer统一编码文本与图像
- 260M模型在ViDoRe v3上达0.523 nDCG@10
- 支持超长上下文，推理速度比ColPali快2倍
多模态模型常基于生成式视觉-语言架构，将独立预训练的视觉编码器与因果语言模型结合。如ColPali等视觉文档检索器复用此类架构作为编码器，为非生成任务带来额外参数与计算开销。本文提出NeoMME，一个包含260M和800M参数的多模态多语言双向编码器家族，使用单个双向Transformer编码器同时处理多语言文本与原始图像块。两个模型均从零开始预训练，采用基于可见图像块的掩码离散扩散文本目标函数。两者均支持16,384词元上下文，足以编码最多两幅标准4K UHD图像。为验证下游能力，我们联合训练密集与晚期交互头。在ViDoRe v3基准上，NeoMME-Retriever 260M优于所有参数量低于800M的模型，nDCG@10达0.523；而NeoMME-Retriever 800M达到0.556。在匹配的2048x2048图像输入下，NVIDIA L40S上，NeoMME-260M的编码吞吐量约为ColModernVBERT的2倍。层级标记池化与非对称量化使晚期交互多模态文档嵌入压缩255倍，同时保持超过95%的基线nDCG@10。我们已将NeoMME贡献至Hugging Face Transformers，并在https://hf.co/collections/Hcompany/neomme下以Apache 2.0协议发布预训练主干与检索兼容检查点。
原文摘要 · Abstract (English)
Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。