arXiv:2607.18666cs.CLcs.SD2026-07被引 3

统一文本图像视频音频的嵌入空间,仅用单卡数小时训练即可

Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

论文配图:Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio
图 1 · 摘自论文原文
  • 用冻结视觉语言模型基座,加轻量连接器和门控适配器融合音频
  • 生成1在无音视频对齐数据下实现音频-图像检索,生成2性能更优
  • 适合需要多模态统一索引的开发者,开源可直接使用

一个覆盖文本、图像、视频和音频的统一嵌入空间,可支持用户提出的任意查询。现有基于视觉-语言骨干网络的嵌入模型在文本/图像/视频检索上领先,但缺乏音频支持;而专门的音频-文本检索系统又无法服务其他模态。本文提出Fusion Embedding系列:在冻结的视觉-语言嵌入基座上,仅训练16.4M参数的连接器(生成1)或加入44.2M参数的模态门控深度适配器(生成2),其分支不作用于文本、图像或视频输入,输出与发布基座完全一致。由于基座已绑定文本、图像和视频,仅对齐音频与文本即可使音频-图像检索能力自然涌现,且无需任何音频-视频成对训练数据。我们通过受控实验验证设计空间——重写训练标题、替换更强音频塔、扩大连接器均导致检索性能下降。训练仅需单张GPU数小时,权重、代码与评估工具均已公开。

原文摘要 · Abstract (English)

A single embedding space that covers text, images, video, and audio lets one index serve every query a user can pose. Embedding models built on vision-language backbones now lead text/image/video retrieval benchmarks but lack audio entirely, while audio-text retrieval is led by specialist systems that serve no other modality. We present the Fusion Embedding family, which adds audio to a frozen vision-language embedding base whose parameters are never updated: generation 1 (fusion-embedding-1) trains only a 16.4M-parameter connector between a frozen audio tower and the frozen base, and generation 2 (fusion-embedding-2) adds modality-gated deep adapters (44.2M parameters) whose branch never executes on text, image, or video inputs: their outputs are bit-for-bit those of the released base, verified after every training run. Because the base already binds text, images, and video, aligning audio to text alone makes audio-image retrieval emerge, with zero paired audio-visual training data. Alongside the recipe we map its design space with controlled negative results (rewriting training captions with an LLM, substituting a leaderboard-stronger audio tower, and widening the connector each reduce retrieval) and with training-protocol findings that we expect to transfer to any frozen decoder-LM embedding backbone. Both generations train in hours on a single GPU. Weights, code, and the evaluation harness are openly released.

多模态嵌入空间音频融合开源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。