提出自适应锚点机制，让多模态模型更均衡地融合文本、图像和音频信息。
Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations
- 用所有模态共同生成可调中心点作为锚点，取代固定单模态锚点。
- 在合成与真实数据集上均超越传统方法，提升多模态对齐效果。
- 适合需要高精度跨模态理解的任务，如图文检索、视频理解。
多模态学习中的统一表示空间对于有效整合文本、图像、音频等异构数据至关重要，能显著提升下游任务的效率与性能。现有绑定方法（如 ImageBind）通常依赖单一固定模态作为锚点，我们从数学上分析发现其存在三大缺陷：(1) 过度依赖锚点模态的选择，(2) 无法充分捕捉模态内信息，(3) 忽视非锚点模态间的跨模态相关性。为此，我们提出自适应锚点绑定方法，以中心点为锚，基于所有可用模态动态生成。所提出的 CentroBind 框架理论上证明能同时实现模态内学习、模态间学习与多模态对齐，构建覆盖全部模态的统一表示空间。在合成与真实数据集上的实验表明，此类自适应方法持续优于固定锚点方法，验证了理论分析。
原文摘要 · Abstract (English)
A unified representation space in multi-modal learning is essential for effectively integrating diverse data sources, such as text, images, and audio, to enhance efficiency and performance across various downstream tasks. Recent binding methods, such as ImageBind, typically rely on a single, fixed anchor modality for aligning multi-modal data. We mathematically analyze these fixed anchor binding methods and uncover significant limitations: (1) over-reliance on the choice of the anchor modality, (2) inadequate capture of intra-modal information, and (3) failure to account for cross-modal correlation among non-anchored modalities. To address these issues, we propose the need for adaptive anchor binding methods, exemplified by our framework CentroBind. The proposed method uses adaptively adjustable centroid-based anchors generated from all available modalities, leading to a balanced and rich representation space. We theoretically demonstrate that our approach captures three critical properties of multi-modal learning -- intra-modal learning, inter-modal learning, and multi-modal alignment -- while constructing a unified representation that spans all modalities. Experiments on both synthetic and real-world datasets show that adaptive anchor methods such as CentroBind consistently outperform fixed anchor binding methods, verifying our analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。