一个统一框架,同时完成声音分离、去混响和声源定位等多重任务。
DeepASA: An Object-Oriented Multi-Purpose Network for Auditory Scene Analysis

- 用对象为中心的思路,将声音特征封装为可追踪的实体
- 在多个基准数据集上达到当前最好性能,尤其在动态场景中表现优异
- 适合需要多任务协同处理的智能音频系统开发
我们提出 DeepASA,一种面向复杂听觉场景的多任务听觉场景分析模型,可在统一框架内完成多输入多输出(MIMO)声源分离、去混响、声音事件检测(SED)、音频分类及到达方向估计(DoAE)。该模型针对时间重叠、空间动态变化且音色相近的多重声源场景设计。通过引入对象导向处理(OOP)策略,将多样听觉特征封装为以对象为中心的表示,并经由链式推理(CoI)机制进行迭代优化。整体架构包含基于动态时间核的特征提取器、基于Transformer的聚合模块以及对象分离器,生成每个对象的专属特征并输入至任务专用解码器。该表示方式自然解决了传统逐轨迹处理中的参数关联模糊问题。为缓解早期对象分离失败对下游任务的影响,我们在链式推理中引入时间一致性匹配(TCM),实现多任务融合与对象特征的持续精炼。在ASA2、MC-FUSS和STARSS23等代表性空间音频基准数据集上的实验表明,DeepASA在所有评估任务中均取得最先进性能,验证了其在多种空间听觉场景下进行声源分离与听觉参数估计的有效性。
原文摘要 · Abstract (English)
We propose DeepASA, a multi-purpose model for auditory scene analysis that performs multi-input multi-output (MIMO) source separation, dereverberation, sound event detection (SED), audio classification, and direction-of-arrival estimation (DoAE) within a unified framework. DeepASA is designed for complex auditory scenes where multiple, often similar, sound sources overlap in time and move dynamically in space. To achieve robust and consistent inference across tasks, we introduce an object-oriented processing (OOP) strategy. This approach encapsulates diverse auditory features into object-centric representations and refines them through a chain-of-inference (CoI) mechanism. The pipeline comprises a dynamic temporal kernel-based feature extractor, a transformer-based aggregator, and an object separator that yields per-object features. These features feed into multiple task-specific decoders. Our object-centric representations naturally resolve the parameter association ambiguity inherent in traditional track-wise processing. However, early-stage object separation can lead to failure in downstream ASA tasks. To address this, we implement temporal coherence matching (TCM) within the chain-of-inference, enabling multi-task fusion and iterative refinement of object features using estimated auditory parameters. We evaluate DeepASA on representative spatial audio benchmark datasets, including ASA2, MC-FUSS, and STARSS23. Experimental results show that our model achieves state-of-the-art performance across all evaluated tasks, demonstrating its effectiveness in both source separation and auditory parameter estimation under diverse spatial auditory scenes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。