解决多语言音频文本检索中的不一致问题，提升跨语言匹配准确率。
ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errors
- 采用1对k对比学习与音视频-英文共锚对比学习
- 在8种主流语言上实现最佳召回率与一致性表现
- 适合需要跨语言音频检索的系统开发者
多语言音频文本检索（ML-ATR）旨在从数据库中检索音频片段或多语言文本。然而，现有方法在跨语言实例相似性匹配中存在不一致问题。本文从多语言模态对齐方向误差和权重误差两方面进行理论分析，提出权重误差上界以量化不一致程度。研究表明，不一致主要源于语言随机采样导致的数据分布误差。为此，提出基于1-to-k对比学习与音频-英文共锚对比学习的一致性方案，有效缓解数据分布误差对召回率和一致性的影响。在翻译后的AudioCaps和Clotho数据集上的实验表明，该方法在包括英语在内的八种主流语言上均达到当前最优的召回率与一致性指标。代码将公开于https://github.com/ATRI-ACL/ATRI-ACL。
原文摘要 · Abstract (English)
Multilingual audio-text retrieval (ML-ATR) is a challenging task that aims to retrieve audio clips or multilingual texts from databases. However, existing ML-ATR schemes suffer from inconsistencies for instance similarity matching across languages. We theoretically analyze the inconsistency in terms of both multilingual modal alignment direction error and weight error, and propose the theoretical weight error upper bound for quantifying the inconsistency. Based on the analysis of the weight error upper bound, we find that the inconsistency problem stems from the data distribution error caused by random sampling of languages. We propose a consistent ML-ATR scheme using 1-to-k contrastive learning and audio-English co-anchor contrastive learning, aiming to mitigate the negative impact of data distribution error on recall and consistency in ML-ATR. Experimental results on the translated AudioCaps and Clotho datasets show that our scheme achieves state-of-the-art performance on recall and consistency metrics for eight mainstream languages, including English. Our code will be available at https://github.com/ATRI-ACL/ATRI-ACL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。