构建统一多模态嵌入模型，解决跨模态融合中的投影漂移问题。
Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding
- 分治融合：先独立训练各模态专家，再融合为统一主干网络。
- 融合后音频检索性能下降，暴露投影器与主干不匹配的漂移问题。
- 通过投影器全参数微调和多模态重训，实现全模态高性能统一。
全模态检索旨在为文本、图像、视频、文档和音频输入建立统一的嵌入空间，但因模态间数据分布、架构和优化动态差异，构建统一检索器极具挑战。本文提出Conan-embedding-v3，采用解耦-融合-恢复框架：首先独立训练各模态专家，再将任务向量融合至单一密集主干，称为解耦专家融合。该策略整合了视觉、视频和文档检索能力，但发现当音频通过外部编码器与投影器接入时，融合主干导致投影器仍适配音频专用主干，引发显著音频检索退化，此现象称投影漂移。为修复此问题，Conan-embedding-v3采用投影器恢复（即冻结主干，对投影器进行全参数微调），并辅以平衡多模态重训。最终模型在单个主干中支持全部检索路径，在MMEB上取得74.9分，在30项任务的MAEB音频套件上达到55.61分。
原文摘要 · Abstract (English)
Omni-modal retrieval promises a single embedding space for text, image, video, document, and audio inputs, but building such a unified retriever is difficult since these modalities differ in data distribution, architecture, and optimization dynamics. In this work, we present Conan-embedding-v3, a decouple--fuse--recover framework for omni-modal retrieval. Conan-embedding-v3 first trains modality specialists independently and fuses their task vectors into a single dense backbone, a strategy we call Decoupled Specialist Fusion. We show that this fusion composes visual, video, and document retrieval capabilities, but also exposes a failure mode for projector-based modalities: when audio is attached through an external encoder and projector, fusing the backbone leaves the projector calibrated to the audio-specialist backbone, causing a large audio retrieval regression despite copying all audio-specific modules unchanged. We call this failure Projector Drift. To repair it, Conan-embedding-v3 applies Projector Recovery (i.e., full-parameter fine-tuning of the projector while keeping the backbone frozen) followed by balanced multi-modal rehearsal. The resulting model supports these retrieval pathways in one backbone, achieving 74.9 scores on MMEB while obtaining 55.61 on the 30-task MAEB audio suite.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。