用高质量数据和轻量模型实现多模态嵌入空间对齐，单卡几小时训练出顶尖性能。
EBind: a practical approach to space binding
- 以单编码器+优质数据为核心，实现高效多模态对齐。
- 18亿参数模型超越4至17倍大模型，关键在三类数据融合。
- 开源代码、模型与数据，推动零样本音频-图文分类评测发展。
我们提出EBind，一种简单、数据驱动且参数高效的多模态嵌入空间对齐方法。通过每模态单一编码器与高质量数据，可在单张GPU上数小时内训练出前沿模型，相比以往需数天的方案显著提速。实验表明，一个1.8B参数的图像-文本-视频-音频-3D联合模型性能超过4至17倍大小的模型。其核心在于三类精心构建的数据：(i) 670万条由先进检索模型自动生成的多模态五元组；(ii) 100万条人工标注的三元组（正/负/部分匹配）；(iii) 340万条已有带字幕数据。我们使用13项评估验证各数据源价值。由于现有基准不足，我们还引入首个高质量、共识标注的音频与图片零样本分类基准。相较此前工作，我们公开全部代码、模型权重与数据集。
原文摘要 · Abstract (English)
We simplify space binding by focusing on two core components, a single encoder per modality and high-quality data; enabling training state-of-the-art models on a single GPU in a few hours as opposed to multiple days. We present EBind, an Easy, data-centric, and parameter-efficient method to Bind the embedding spaces of multiple contrastive models. We demonstrate that a simple 1.8B-parameter image-text-video-audio-3D model can outperform models 4 to 17x the size. The key to achieving this is a carefully curated dataset of three complementary data sources: i) 6.7M fully-automated multimodal quintuples sourced via SOTA retrieval models, ii) 1M diverse, semi-automated triples annotated by humans as negative, partial, or positive matches, and iii) 3.4M pre-existing captioned data items. We use 13 different evaluations to demonstrate the value of each data source. Due to limitations with existing benchmarks, we further introduce the first high-quality, consensus-annotated zero-shot classification benchmark between audio and PCs. In contrast to related work, we will open-source our code, model weights, and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。