让声音检测能识别任意新类别，靠文本和音频提示实现开集检测。
Detect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries
- 用多模态提示生成查询向量，将声音检测转为帧级匹配任务。
- 在 AudioSet 强标注数据集上，开集检测性能比 CLAP 提升 7.8 PSDS。
- 无需训练即可识别新类别，适合需要灵活扩展的应用场景。
现有声音事件检测（SED）方法大多基于封闭集假设，仅能识别预定义类别。尽管近期研究尝试通过语言驱动实现零样本检测，但受限于细粒度对齐和跨模态特征融合不足，性能仍不理想。本文提出 Detect Any Sound Model（DASM），一种基于多模态查询的开集声音事件检测框架。DASM 将 SED 定义为帧级检索任务，将音频特征与来自文本或音频提示的查询向量进行匹配。为此，DASM 设计双流解码器：跨模态事件解码器实现查询-特征融合并判断片段级事件是否存在，上下文网络建模时序依赖以实现帧级定位。此外，引入推理阶段注意力掩码策略，利用基础类与新类间的语义关联，显著提升对新类别的泛化能力。在 AudioSet Strong 数据集上的实验表明，DASM 在开集设置下性能优于 CLAP 方法 7.8 PSDS，闭集设置下优于基线 6.9 PSDS。在 DESED 数据集上的跨数据集零样本评估中，DASM 达到 42.2 PSDS1，甚至超过监督训练的 CRNN 基线。
原文摘要 · Abstract (English)
Most existing sound event detection~(SED) algorithms operate under a closed-set assumption, restricting their detection capabilities to predefined classes. While recent efforts have explored language-driven zero-shot SED by exploiting audio-language models, their performance is still far from satisfactory due to the lack of fine-grained alignment and cross-modal feature fusion. In this work, we propose the Detect Any Sound Model (DASM), a query-based framework for open-vocabulary SED guided by multi-modal queries. DASM formulates SED as a frame-level retrieval task, where audio features are matched against query vectors derived from text or audio prompts. To support this formulation, DASM introduces a dual-stream decoder that explicitly decouples event recognition and temporal localization: a cross-modality event decoder performs query-feature fusion and determines the presence of sound events at the clip-level, while a context network models temporal dependencies for frame-level localization. Additionally, an inference-time attention masking strategy is proposed to leverage semantic relations between base and novel classes, substantially enhancing generalization to novel classes. Experiments on the AudioSet Strong dataset demonstrate that DASM effectively balances localization accuracy with generalization to novel classes, outperforming CLAP-based methods in open-vocabulary setting (+ 7.8 PSDS) and the baseline in the closed-set setting (+ 6.9 PSDS). Furthermore, in cross-dataset zero-shot evaluation on DESED, DASM achieves a PSDS1 score of 42.2, even exceeding the supervised CRNN baseline. The project page is available at https://cai525.github.io/Transformer4SED/demo_page/DASM/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。