arXiv:2604.11043cs.AI2026-04KDD

让跨模态模型在无配对数据时也能精准识别,无需额外标注。

EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models

论文配图:EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models
图 1 · 摘自论文原文
  • 通过构建噪声桥接锚点,避免梯度干扰。
  • 在9个数据集上零样本分类与检索性能均超越基线。
  • 适合希望少标注就能扩展多模态系统的研究者。

统一的多模态嵌入空间支撑着跨模态检索和零样本识别等实际应用。但在许多真实场景中,仅部分模态对(如图像-文本)有监督数据,导致其他未配对模态对(如音频-深度、红外-音频)连接薄弱,零样本迁移表现差。为解决这一稀疏配对问题,本文提出【EmergentBridge】——一种无需完整配对标注的嵌入级桥接框架。核心观察发现:将新模态直接对齐合成代理嵌入会引入梯度干扰,破坏现有检索/分类依赖的锚点对齐结构。EmergentBridge通过两步解决:(i) 学习从锚点嵌入生成一个‘噪声桥接锚点’(已有对齐模态的代理嵌入);(ii) 仅在与锚点对齐方向正交的子空间中强制代理对齐,从而保持锚点对齐的同时增强非锚点间的连通性。在涵盖多个模态的9个数据集上,EmergentBridge在零样本分类与检索任务中持续优于先前基线,展现出强大的涌现对齐能力。

原文摘要 · Abstract (English)

Unified multimodal embedding spaces underpin practical applications such as cross-modal retrieval and zero-shot recognition. In many real deployments, however, supervision is available only for a small subset of modality pairs (e.g., image--text), leaving \emph{unpaired} modality pairs (e.g., audio$\leftrightarrow$depth, infrared$\leftrightarrow$audio) weakly connected and thus performing poorly on zero-shot transfer. Addressing this sparse-pairing regime is therefore essential for scaling unified embedding systems to new tasks without curating exhaustive pairwise data. We propose \textbf{EmergentBridge}, an embedding-level bridging framework that improves performance on these unpaired pairs \emph{without requiring exhaustive pairwise supervision}. Our key observation is that naively aligning a new modality to a synthesized proxy embedding can introduce \emph{gradient interference}, degrading the anchor-alignment structure that existing retrieval/classification relies on. EmergentBridge addresses this by (i) learning a mapping that produces a \emph{noisy bridge anchor} (a proxy embedding of an already-aligned modality) from an anchor embedding, and (ii) enforcing proxy alignment only in the subspace orthogonal to the anchor-alignment direction, preserving anchor alignment while strengthening non-anchor connectivity. Across nine datasets spanning multiple modalities, EmergentBridge consistently outperforms prior binding baselines on zero-shot classification and retrieval, demonstrating strong emergent alignment.

多模态零样本嵌入对齐桥接

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。