arXiv:2601.03666cs.CLcs.AI2026-01ACL被引 11

提出显式对齐方法,让多模态嵌入更稳定准确

e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings

  • 用温度校准、负样本调度和批归一化提升跨模态对齐
  • 在MMEB-V2和AudioCaps上优于现有模型,性能提升显著
  • 轻量改造适配多种视觉语言模型,适合多模态应用开发

现代信息系统常涉及文本、图像、视频、音频等异构数据,推动了多模态嵌入模型的发展,旨在将不同模态映射到共享空间以实现直接比较。然而,现有模型仍严重依赖预训练视觉-语言模型(VLM)的隐式对齐,导致三个问题:(i) 相似度分数的模态相关锐度不一致,评分尺度不统一;(ii) 混合模态批次中负样本难度分布失衡,许多负样本迅速变得简单,梯度贡献小;(iii) 跨模态嵌入的一阶与二阶统计特性不匹配,影响排序稳定性。为此,我们提出e5-omni,一种轻量级显式对齐方案,可将现成VLM转化为鲁棒的多模态嵌入模型。该方法包含三部分:(1) 模态感知温度校准以对齐相似度尺度;(2) 可控负样本课程学习与去偏机制,聚焦难例负样本并降低误负样本影响;(3) 批归一化结合协方差正则化,更好匹配共享嵌入空间中的跨模态几何结构。在MMEB-V2和AudioCaps上的实验表明,e5-omni持续优于强基线,且该方案可良好迁移至其他VLM骨干网络。模型权重已发布于https://huggingface.co/Haon-Chen/e5-omni-7B。

原文摘要 · Abstract (English)

Modern information systems often involve different types of items, e.g., a text query, an image, a video clip, or an audio segment. This motivates omni-modal embedding models that map heterogeneous modalities into a shared space for direct comparison. However, most recent omni-modal embeddings still rely heavily on implicit alignment inherited from pretrained vision-language model (VLM) backbones. In practice, this causes three common issues: (i) similarity logits have modality-dependent sharpness, so scores are not on a consistent scale; (ii) in-batch negatives become less effective over time because mixed-modality batches create an imbalanced hardness distribution; as a result, many negatives quickly become trivial and contribute little gradient; and (iii) embeddings across modalities show mismatched first- and second-order statistics, which makes rankings less stable. To tackle these problems, we propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf VLMs into robust omni-modal embedding models. e5-omni combines three simple components: (1) modality-aware temperature calibration to align similarity scales, (2) a controllable negative curriculum with debiasing to focus on confusing negatives while reducing the impact of false negatives, and (3) batch whitening with covariance regularization to better match cross-modal geometry in the shared embedding space. Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines, and the same recipe also transfers well to other VLM backbones. We release our model checkpoint at https://huggingface.co/Haon-Chen/e5-omni-7B.

多模态嵌入显式对齐VLM跨模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。