用视听线索分离混响音源，提升XR中语音可懂度与交互体验
MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR
- 视听协同分步分离：先音频粗分，再用视觉目标引导精调
- 支持最多5路并发音源，延迟约2秒，语音理解提升36.2%
- 适合需要高声学感知的虚拟会议、沉浸式演出等场景
在扩展现实（XR）中，复杂的声学环境常因声源混叠而影响用户对场景的感知与社交参与。我们提出MoXaRt，一种基于视听线索的实时XR系统，可分离复杂声源并实现细粒度声音交互。其核心为级联架构：并行执行仅音频的粗分离，同时通过视觉检测（如人脸、乐器）定位声源；这些视觉锚点随后指导精修网络，实现单个声源的分离，可处理最多5路并发声源（如2人说话+3种乐器），处理延迟约为2秒。我们在新构建的30段一分钟录音数据集上进行技术评估，并开展22名参与者用户研究。结果表明，系统显著提升语音可懂度，在对抗性声学环境中听觉理解能力提升36.2%（p < 0.01），同时大幅降低认知负荷（p < 0.001），为更智能、更具社交感知力的XR体验铺平道路。
原文摘要 · Abstract (English)
In Extended Reality (XR), complex acoustic environments often overwhelm users, compromising both scene awareness and social engagement due to entangled sound sources. We introduce MoXaRt, a real-time XR system that uses audio-visual cues to separate these sources and enable fine-grained sound interaction. MoXaRt's core is a cascaded architecture that performs coarse, audio-only separation in parallel with visual detection of sources (e.g., faces, instruments). These visual anchors then guide refinement networks to isolate individual sources, separating complex mixes of up to 5 concurrent sources (e.g., 2 voices + 3 instruments) with ~2 second processing latency. We validate MoXaRt through a technical evaluation on a new dataset of 30 one-minute recordings featuring concurrent speech and music, and a 22-participant user study. Empirical results indicate that our system significantly enhances speech intelligibility, yielding a 36.2% (p < 0.01) increase in listening comprehension within adversarial acoustic environments while substantially reducing cognitive load (p < 0.001), thereby paving the way for more perceptive and socially adept XR experiences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。