无监督训练下，像素也能听懂多语言语音。
VeS: Teaching Pixels to Listen Without Supervision
- 用密集匹配替代全局池化，提升跨模态对齐
- 在印度多语言数据上，音频视觉检索准确率提升59%
- 无需微调视觉模型，即可实现精准零样本定位
现有密集音视频模型在英语主导、带字幕的网络视频中表现优异，但其在低资源、多语种混杂、噪声大的发展中国家场景中是否有效尚不明确。我们以涵盖数十种印度语言及方言的Project Vaani多语言子集为测试基准，对比三种对比学习目标：(i) 全局均值池化损失（类CLIP），(ii) 密集最大-均值标记匹配（类DenseAV），(iii) 一种受冻结视觉对齐启发的简单混合方案。结果显示，密集匹配相比全局池化在音频视觉检索任务中实现59%相对提升，平均与中位排名显著降低，且在保持视觉主干完全冻结（无LoRA或部分微调）的前提下，仍能生成清晰的零样本定位热图。结果表明，在标注稀缺、声学环境嘈杂时，密集标记路由更为关键。代码与训练模型已开源。
原文摘要 · Abstract (English)
Recent dense audio-visual (AV) models achieve impressive retrieval and emergent localization, but almost all evidence comes from English-centric, caption-rich web video. It is unclear whether these objectives survive in low-resource, code-switched, and noisy multilingual settings that typify developing regions. We show they do**-**and that the choice of aggregation function becomes even more critical. Using a multilingual subset of Project Vaani spanning dozens of Indian languages and dialectal variants, we compare three contrastive objectives: (i) a global mean-pooled loss (CLIP-style), (ii) a dense max-mean token matcher (DenseAV-style), and (iii) a simple hybrid (motivated by frozen-vision alignment strategies). The dense objective delivers a +59% relative R@1 (Audio Visual) improvement over global pooling and substantially lower mean/median ranks, while consistently producing sharp zero-shot localization heatmaps of spoken objects-despite keeping the vision backbone entirely frozen (no LoRA / partial fine-tuning). Our results demonstrate that dense token routing is not a luxury of high-resource English corpora; it is more decisive when annotations and acoustic cleanliness are scarce. We release the codebase and trained models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。