用音频描述的相似性推断非二元相关性,提升文本检索音频效果。
Text-based Audio Retrieval by Learning from Similarities between Audio Captions
- 通过句向量相似度计算音频描述间关系,生成非二元相关分数
- 在Clotho数据集上,相比二值化方法,检索准确率提升5.2%(mAP)
- 适合构建细粒度文本到音频检索系统的研究者使用
本文提出利用音频描述间的相似性来估计音频-描述相关性,用于训练文本驱动的音频检索系统。现有音频-描述数据集(如Clotho)虽包含配对样本,但缺乏除标注对之外的显式相关性信息。主流方法(如CLAP)通常将标注对视为正例,其余组合全为负例,假设音频与描述间仅存在二元相关。为推断任意描述与音频间的相关性,本文提出一种方法:基于音频描述的文本相似度(通过Sentence-BERT嵌入的余弦相似度计算),经逻辑函数转换生成非二元相关分数,从而将音频样本通过其标注描述关联至数据集中所有其他描述。为融入训练,采用列表级排序目标,将相关分数转化为给定文本查询下音频样本的排序概率。实验表明,该方法在文本驱动音频检索任务中优于使用二元相关性的基线模型。
原文摘要 · Abstract (English)
This paper proposes to use similarities of audio captions for estimating audio-caption relevances to be used for training text-based audio retrieval systems. Current audio-caption datasets (e.g., Clotho) contain audio samples paired with annotated captions, but lack relevance information about audio samples and captions beyond the annotated ones. Besides, mainstream approaches (e.g., CLAP) usually treat the annotated pairs as positives and consider all other audio-caption combinations as negatives, assuming a binary relevance between audio samples and captions. To infer the relevance between audio samples and arbitrary captions, we propose a method that computes non-binary audio-caption relevance scores based on the textual similarities of audio captions. We measure textual similarities of audio captions by calculating the cosine similarity of their Sentence-BERT embeddings and then transform these similarities into audio-caption relevance scores using a logistic function, thereby linking audio samples through their annotated captions to all other captions in the dataset. To integrate the computed relevances into training, we employ a listwise ranking objective, where relevance scores are converted into probabilities of ranking audio samples for a given textual query. We show the effectiveness of the proposed method by demonstrating improvements in text-based audio retrieval compared to methods that use binary audio-caption relevances for training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。