让图文模型在任意模态组合下都稳定表现。
UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddings
- 用文本生成视觉特征，补全缺失模态。
- 跨模态对齐训练使多模态嵌入更鲁棒。
- 适合处理稀有模态组合的复杂查询场景。
现有视觉-语言模型在信息检索等多模态嵌入任务中表现良好，但在涉及多样化模态组合的真实查询与目标中仍面临挑战。原因在于，传统方法在训练时无法统一对齐所有模态组合，导致推理时对罕见模态模式性能下降。为此，我们提出UniMoCo，其核心是一个模态补全模块，可从文本生成视觉特征，确保查询与目标在嵌入空间中的模态完整性。同时，采用特殊训练策略，对原始输入与模态补全后的输入进行联合对齐，实现多样模态组合下的一致且鲁棒的嵌入表示。大量实验表明，UniMoCo优于以往方法，并在多种设置下保持稳定性能。我们还量化了传统方法因训练数据中模态组合不平衡带来的固有偏差，证实该模态补全范式能有效缓解此问题。代码已开源：https://github.com/HobbitQia/UniMoCo。
原文摘要 · Abstract (English)
Current vision-language models have been explored for multi-modal embedding tasks like information retrieval. However, they face significant challenges in real-world queries and targets involving diverse modality combinations, as existing approaches often fail to align all modality combinations within a unified embedding space during training, leading to degraded performance on rare modality patterns during inference. To address this fundamental limitation, we propose UniMoCo, a novel architecture featuring a modality-completion module that generates visual features from text, thereby ensuring modality completeness for both queries and targets. Additionally, UniMoCo incorporates a specialized training strategy that aligns embeddings from both original and modality-completed inputs, thus ensuring consistent and robust embeddings for diverse modality combinations. Comprehensive experiments demonstrate that UniMoCo outperforms previous methods while exhibiting consistent robustness across diverse settings. Furthermore, we identify and quantify the inherent bias in conventional approaches caused by imbalanced modality combinations in training data, showing that our modality-completion paradigm effectively mitigates this limitation. The code is available at https://github.com/HobbitQia/UniMoCo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。