用分治代码本实现多模态特征对齐，提升跨模态理解能力
CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook
- 将特征拆分为共享与专属部分，分别对应语义一致性和模态特异性
- 在9种模态上达成当前最优分类与检索性能
- 无需完全配对数据，适合数据稀缺场景下的多模态学习
多模态表征对齐对大语言模型和机器人技术至关重要。传统方法常受跨模态信息差异和数据稀缺制约，导致对齐空间表现不佳，忽略模态独有特征。我们提出CodeBind框架，通过设计模态共享-专属代码本优化多模态表征空间。该方法分步对齐目标模态与桥梁模态，无需完整配对数据。不同于传统硬对齐，CodeBind将特征分解为共享成分（保障语义一致性）与专属成分（保留模态特异性）。采用组合式向量量化机制：共享代码本弥合模态差距，专属代码本缓解表示偏倚，防止主导模态压制其他模态。在文本、图像、视频、音频、深度图、热成像、触觉、3D点云、脑电（EEG）共九种模态上验证，CodeBind在多模态分类与检索任务中达到当前最优性能。
原文摘要 · Abstract (English)
Multimodal representation alignment is pivotal for large language models and robotics. Traditional methods are often hindered by cross-modal information discrepancies and data scarcity, leading to suboptimal alignment spaces that overlook modality-unique features. We propose CodeBind, a framework that optimizes multimodal representation spaces through a modality-shared-specific codebook design. By incrementally aligning target and bridging modalities, CodeBind bypasses the need for fully paired data. Unlike traditional hard alignment, CodeBind decomposes features into shared components for semantic consistency and specific components for modality-unique details. This design utilizes a compositional vector quantization scheme, where a shared codebook bridges modality gaps and modality-specific codebooks mitigate representation bias by preventing dominant modalities from overshadowing others. Validated across nine modalities (text, image, video, audio, depth, thermal, tactile, 3D point cloud, EEG), CodeBind achieves state-of-the-art performance in multimodal classification and retrieval tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。