用注意力机制让文字和音频对齐，提升语音检索准确率
Language-based Audio Retrieval with Co-Attention Networks
- 设计级联注意力网络，逐步优化文本与音频的语义匹配
- 在Clotho和AudioCaps数据集上分别提升16.6%和15.1%的检索精度
- 适合需要跨模态检索的语音应用开发者参考
近年来，用户生成的音频内容在各类媒体平台迅速增长，对基于自然语言查询的高效音频检索方法需求日益迫切。该任务面临的主要挑战在于，如何从文本和音频两种异构模态中学习有效的语义表示。本文提出一种新的语言-音频检索框架，利用协同注意力机制联合学习双模态的语义表示。为增强模型捕捉细粒度跨模态交互的能力，我们设计了级联式协同注意力结构，通过堆叠或迭代多个注意力模块，逐步优化文本与音频之间的语义对齐。在两个公开数据集上的实验表明，所提方法优于当前最优模型：在Clotho数据集上，平均精确率（mean Average Precision）提升16.6%；在AudioCaps数据集上提升15.1%。
原文摘要 · Abstract (English)
In recent years, user-generated audio content has proliferated across various media platforms, creating a growing need for efficient retrieval methods that allow users to search for audio clips using natural language queries. This task, known as language-based audio retrieval, presents significant challenges due to the complexity of learning semantic representations from heterogeneous data across both text and audio modalities. In this work, we introduce a novel framework for the language-based audio retrieval task that leverages co-attention mechanismto jointly learn meaningful representations from both modalities. To enhance the model's ability to capture fine-grained cross-modal interactions, we propose a cascaded co-attention architecture, where co-attention modules are stacked or iterated to progressively refine the semantic alignment between text and audio. Experiments conducted on two public datasets show that the proposed method can achieve better performance than the state-of-the-art method. Specifically, our best performed co-attention model achieves a 16.6% improvement in mean Average Precision on Clotho dataset, and a 15.1% improvement on AudioCaps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。