用深度学习改进头转动时的立体声信号匹配,减少听感失真。
SpatialNet with Binaural Loss Function for Correcting Binaural Signal Matching Outputs under Head Rotations
- 融合SpatialNet网络与双耳感知损失函数,提升空间信息处理能力。
- 在六麦克风半圆阵列上验证,头转动时仍保持信号准确性。
- 适合虚拟现实、智能眼镜等需要精准立体声的可穿戴设备场景。
随着虚拟现实头显、智能眼镜和头动追踪耳机的兴起,立体声再现日益重要。这些设备常使用任意布置的麦克风阵列,空间分辨率有限,导致立体声信号准确度下降,尤其在头部旋转时。已有方法BSM-MagLS虽改善了高频重现和头动情况下的表现,但头部转动越大,信号误差越明显,出现空间与音色失真,特别是虚拟听者耳朵远离最近麦克风时。本文提出将深度学习与BSM-MagLS结合,在后处理阶段引入基于SpatialNet的框架,利用其对空间信息的有效处理能力,并同时优化信号级损失和基于人类双耳听觉理论的感知性双耳损失。仿真研究采用六麦克风半圆阵列,证明该方法在不同头动角度下均具鲁棒性。进一步的听觉实验在多种混响声学环境中进行,结果表明该框架能有效缓解BSM-MagLS的退化问题,在大幅头动下仍提供稳定修正。
原文摘要 · Abstract (English)
Binaural reproduction is gaining increasing attention with the rise of devices such as virtual reality headsets, smart glasses, and head-tracked headphones. Achieving accurate binaural signals with these systems is challenging, as they often employ arbitrary microphone arrays with limited spatial resolution. The Binaural Signals Matching with Magnitude Least-Squares (BSM-MagLS) method was developed to address limitations of earlier BSM formulations, improving reproduction at high frequencies and under head rotation. However, its accuracy still degrades as head rotation increases, resulting in spatial and timbral artifacts, particularly when the virtual listener's ear moves farther from the nearest microphones. In this work, we propose the integration of deep learning with BSM-MagLS to mitigate these degradations. A post-processing framework based on the SpatialNet network is employed, leveraging its ability to process spatial information effectively and guided by both signal-level loss and a perceptually motivated binaural loss derived from a theoretical model of human binaural hearing. The effectiveness of the approach is investigated in a simulation study with a six-microphone semicircular array, showing its ability to perform robustly across head rotations. These findings are further studied in a listening experiment across different reverberant acoustic environments and head rotations, demonstrating that the proposed framework effectively mitigates BSM-MagLS degradations and provides robust correction across substantial head rotations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。