提出融合教师蒸馏方法，实现音视频文本任意组合检索。
OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation
- 用融合嵌入作教师信号，指导单模态模型学习联合表示。
- 在6个零样本测试集上超越Gemini Embedding 2，最高提升18.0。
- 适合需要跨模态联合检索的开放研究者使用。
统一多模态嵌入空间已成为跨模态检索和多模态RAG的标准接口，近期的音视频文本（AVT）编码器将此扩展至三模态。此类编码器可在三模态均可用时生成联合（T,V,A）嵌入，但标准成对InfoNCE目标函数在训练中未利用该信号。本文提出融合作为教师蒸馏（fusion-as-teacher distillation），将带停止梯度的融合嵌入视为教师信号，用于监督单模态嵌入，并引入元组型InfoNCE项直接监督融合嵌入。我们基于此构建OmniRetriever-7B。在六个零样本检索基准上，OmniRetriever-7B在Clotho和SoundDescs上较闭源Gemini Embedding 2提升13.3–18.0 R@1，达到当前开放视频-文本编码器的零样本顶尖水平。为压力测试联合表示，我们发布OmniRetriever-Bench，一个包含3782个三元组的12方向音视频文本检索基准；在该基准上，OmniRetriever-7B平均全向得分34.84，较Gemini Embedding 2提升1.72，较最佳开源先验方法提升8.03。
原文摘要 · Abstract (English)
Unified multimodal embedding spaces have become the standard interface for cross-modal retrieval and multimodal RAG, and recent audio-video-text (AVT) encoders extend this setting to three modalities. Such encoders can produce a joint (T,V,A) embedding whenever all three modalities are available, but standard pairwise InfoNCE objectives leave this signal unused during training. We close this gap with fusion-as-teacher distillation, which treats a stop-gradient copy of the fused embedding as a teacher signal for the single-modal embeddings, paired with a Tuple-InfoNCE term that supervises the fused embedding directly. We instantiate this objective as OmniRetriever-7B. Across six zero-shot retrieval benchmarks, OmniRetriever-7B surpasses the closed-source Gemini Embedding 2 by 13.3-18.0 R@1 on Clotho and SoundDescs, and reaches the contemporary zero-shot specialist band of open video-text encoders on MSR-VTT and MSVD. To stress-test joint representations, we further release OmniRetriever-Bench, a 12-direction AVT retrieval benchmark totaling 3782 triples; on it OmniRetriever-7B attains AVG-all 34.84, improving over Gemini Embedding 2 by 1.72 and over the best prior open-source AVT method by 8.03.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。