让模型从长音频中精准定位文本描述的片段，提升语音检索精度。
Language-based Audio Moment Retrieval
- 基于DETR架构设计时序模型，捕捉长音频内部依赖关系。
- 在模拟数据集上，召回率最高提升9.00点，优于传统滑窗方法。
- 提供真实标注数据集，适合研究语音与语言对齐的学者使用。
本文提出并构建了一项新任务——音频片段检索（AMR）。与传统基于文本的短音频检索不同，AMR旨在根据文本查询，在未剪辑的长音频中预测相关时间片段。由于该任务尚无先例，我们首先构建了大规模模拟音频数据集Clotho-Moment，包含带片段标注的音频。随后提出一种基于DETR的模型——音频片段DETR（AM-DETR），通过建模音频特征中的时序依赖性，超越传统的逐片段检索方法。此外，我们还提供了人工标注的真实数据集，用于评估方法的有效性和鲁棒性。实验表明，AM-DETR在训练后各项指标均优于基于滑动窗口的基线模型，尤其在[email protected]上提升9.00点。相关数据集与代码已公开。
原文摘要 · Abstract (English)
In this paper, we propose and design a new task called audio moment retrieval (AMR). Unlike conventional language-based audio retrieval tasks that search for short audio clips from an audio database, AMR aims to predict relevant moments in untrimmed long audio based on a text query. Given the lack of prior work in AMR, we first build a dedicated dataset, Clotho-Moment, consisting of large-scale simulated audio recordings with moment annotations. We then propose a DETR-based model, named Audio Moment DETR (AM-DETR), as a fundamental framework for AMR tasks. This model captures temporal dependencies within audio features, inspired by similar video moment retrieval tasks, thus surpassing conventional clip-level audio retrieval methods. Additionally, we provide manually annotated datasets to properly measure the effectiveness and robustness of our methods on real data. Experimental results show that AM-DETR, trained with Clotho-Moment, outperforms a baseline model that applies a clip-level audio retrieval method with a sliding window on all metrics, particularly improving [email protected] by 9.00 points. Our datasets and code are publicly available in https://h-munakata.github.io/Language-based-Audio-Moment-Retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。