让声音检测支持自由文本查询,实现零样本和少样本识别
FlexSED: Towards Open-Vocabulary Sound Event Detection
- 用音频自监督模型+文本编码器构建端到端框架
- 在AudioSet-Strong上性能超越传统模型,支持零样本与少样本
- 借助大语言模型辅助选题,解决标签缺失问题
尽管大规模声音事件检测(SED)系统已能处理数百种声音类别,现有多分类框架仍存在根本局限:无法处理自由文本声音查询,缺乏零样本能力,且少样本适应性差。虽已有基于文本查询的分离方法,但主要针对声源分离,不适用于需要精确时间定位和跨大规模多样化声音词汇的SED任务。本文提出FlexSED,一种开放词汇声音事件检测系统。FlexSED基于预训练音频自监督模型和CLAP文本编码器,引入编码器-解码器结构与自适应融合策略,实现从预训练权重的有效连续训练。为确保稳健监督,还利用大语言模型(LLMs)在训练中协助选择事件查询,缓解标签缺失问题。结果表明,FlexSED在AudioSet-Strong上性能优于基线SED模型,并展现出强大的零样本与少样本能力。代码与预训练模型已开源,以支持后续研究与应用。
原文摘要 · Abstract (English)
Despite recent progress in large-scale sound event detection (SED) systems capable of handling hundreds of sound classes, existing multi-class classification frameworks remain fundamentally limited. They cannot process free-text sound queries, which enable more flexible and user-friendly interaction, and they lack zero-shot capabilities and offer poor few-shot adaptability. Although text-query-based separation methods have been explored, they primarily focus on source separation and are ill-suited for SED tasks that require precise temporal localization and efficient detection across large and diverse sound vocabularies. In this paper, we propose FlexSED, an open-vocabulary sound event detection system. FlexSED builds on a pretrained audio SSL model and the CLAP text encoder, introducing an encoder-decoder composition and an adaptive fusion strategy to enable effective continuous training from pretrained weights. To ensure robust supervision, it also employs large language models (LLMs) to assist in event query selection during training, addressing challenges related to missing labels. As a result, FlexSED achieves superior performance compared to vanilla SED models on AudioSet-Strong, while demonstrating strong zero-shot and few-shot capabilities. We release the code and pretrained models to support future research and applications based on FlexSED.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。