从真实环境音频中提取时空隐含表示,识别不同声学场景差异。
Spatio-temporal Latent Representations for the Analysis of Acoustic Scenes in-the-wild
- 用自监督学习将音频事件转为嵌入,构建时空隐空间。
- 在地铁与室内场景间发现显著声学差异,隐空间分布清晰可分。
- 适合做声学场景分析、异常检测的科研人员参考。
在声学场景分析领域,本文提出一种新方法,从真实环境音频数据中提取时空隐含表示。基于自建数据集WE-LIVE(包含多样真实环境录音、稀疏GPS坐标及自标注情感与情境标签),以音频段落定位对应位置作为预训练任务,最终目标为声学检测暴力等异常情境(留待后续工作)。通过YAMNet模型在AudioSet上训练,实现对WE-LIVE中音频事件的时序定位与识别;再将离散事件转换为嵌入,比较基于信息检索的TF-IDF算法与类自然语言处理的Node2Vec方法。随后训练变分自编码器(VAE)生成适配的隐空间。通过余弦距离测量与t-SNE可视化分析,揭示出不同声学场景的结构差异。尤其在地铁与室内环境中,隐空间内数据分布明显分离,而编码前数据点呈随机分布。本研究首次实现从真实环境音频中有效提取具有语义意义的时空隐表示。
原文摘要 · Abstract (English)
In the field of acoustic scene analysis, this paper presents a novel approach to find spatio-temporal latent representations from in-the-wild audio data. By using WE-LIVE, an in-house collected dataset that includes audio recordings in diverse real-world environments together with sparse GPS coordinates, self-annotated emotional and situational labels, we tackle the challenging task of associating each audio segment with its corresponding location as a pretext task, with the final aim of acoustically detecting violent (anomalous) contexts, left as further work. By generating acoustic embeddings and using the self-supervised learning paradigm, we aim to use the model-generated latent space to acoustically characterize the spatio-temporal context. We use YAMNet, an acoustic events classifier trained in AudioSet to temporally locate and identify acoustic events in WE-LIVE. In order to transform the discrete acoustic events into embeddings, we compare the information-retrieval-based TF-IDF algorithm and Node2Vec as an analogy to Natural Language Processing techniques. A VAE is then trained to provide a further adapted latent space. The analysis was carried out by measuring the cosine distance and visualizing data distribution via t-Distributed Stochastic Neighbor Embedding, revealing distinct acoustic scenes. Specifically, we discern variations between indoor and subway environments. Notably, these distinctions emerge within the latent space of the VAE, a stark contrast to the random distribution of data points before encoding. In summary, our research contributes a pioneering approach for extracting spatio-temporal latent representations from in-the-wild audio data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。