抖音多模态嵌入模型融合对比学习与推理机制,实现高效精准匹配。
Douyin Multimodal Embedding Model Technical Report

- 分两阶段训练:先大规模对比预训练,再通过隐式推理和跨向重建提升语义精度。
- 在MMEB-v2上2B和9B版本分别达74.8和78.4,视频与图文任务表现突出。
- 部署于抖音全场景,线上搜索带来0.1%生命周期收益,适合工业级多模态应用。
多模态表示学习是现代AI的核心。通过将多模态查询与目标编码为向量,支撑工业级搜索推荐并赋能智能体。抖音、小红书、YouTube等复杂模态、海量内容的平台,要求在百亿级索引下兼具效率与细粒度匹配能力。现有多模态大模型(MLLM)难以兼顾两者:对比模型高效但监督粒度粗,基于思维链(CoT)的模型虽提升区分度,却难以在线服务。我们提出抖音多模态嵌入(DME),分两阶段训练以融合双重优势。第一阶段进行大规模对比预训练,建立覆盖广泛模态与任务的统一嵌入空间;第二阶段通过两个机制增强语义完备性——证据感知的类型化隐式推理(Evidence-Grounded Typed Latent Reasoning)组织检索证据,跨条件重构(Cross-Conditional Reconstruction)通过双向自回归重构强制保留对应侧语义。二者仅在训练中生效,对查询端开销极小,使DME在线服务效率等同标准对比编码器。在MMEB-v2基准上,2B与9B版本分别达到74.8与78.4,性能领先;生产环境中,于抖音内部离线评估集提升2.92%,已部署于生成、图像、AI搜索等场景,线上搜索测试中带来0.1%生命周期(LT)增长。
原文摘要 · Abstract (English)
Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。