一个模型同时生成文本和多模态的稠密与稀疏向量,提升检索效率。
UEmbed: Unified Sparse and Dense Multimodal Embeddings

- 用解码器架构在单次前向传播中生成稠密和稀疏嵌入。
- UEmbed-9B在MMEB-v2上稠密和稀疏指标分别达71.8和71.0。
- 适合需要高效多模态检索的系统或智能体应用。
稀疏检索支撑现代搜索系统,从网络搜索到检索增强生成。现有工作引入学习型稀疏检索(LSR)以超越精确词匹配,实现更丰富的语义。然而LSR仍依赖编码器式双向结构,且多模态扩展仍严重依赖辅助跨模态模块。为此,我们提出UEmbed(统一嵌入),一种仅用解码器的多模态嵌入模型,可在一次因果前向传播中生成稀疏词法和稠密表示。UEmbed在输入后附加N个可学习特殊标记,并将词表划分为N个互不重叠子集。每个标记的因果隐藏状态预测其分配子集上的稀疏权重,再将N个子集拼接成完整稀疏向量。在公开数据上训练,我们发布2B、4B和9B三种规模的UEmbed。UEmbed-9B在MMEB-v2上达到71.8(稠密)和71.0(稀疏),优于其他基于公开数据训练的多模态嵌入模型(如RzenEmbed)。在BEIR上,其表现也优于强稠密和稀疏基线。此外,我们在有效性、效率和智能体应用三方面验证了UEmbed的实用性。总体而言,UEmbed提供新范式:统一稠密与稀疏嵌入于单一模型,并将稀疏检索拓展至文本与多模态输入的统一处理。
原文摘要 · Abstract (English)
Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to multimodal settings still relies heavily on auxiliary cross-modal modules. To address these limitations, we introduce UEmbed (Unified Embedding), a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets. Each token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, outperforming multimodal embedding models trained on publicly available data (e.g., RzenEmbed). On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications. Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。