用背景音乐无感测姿,突破传统声波测姿的侵入性限制。
BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Sounds
- 利用自然音乐作为主动传感信号,避免传统鸣响信号的不适感。
- 通过对比学习与频段注意力机制,从混杂音乐中提取微弱姿态信息。
- 适用于日常场景下的无感人体姿态捕捉,适合智能交互与可穿戴应用。
我们提出 BGM2Pose,一种利用任意音乐(如背景音乐)作为主动感知信号的非侵入式三维人体姿态估计方法。与依赖可听范围内侵入性啁啾信号而严重限制实用性的现有方法不同,本方法采用对人类几乎无不适感的自然音乐。从标准音乐中估计人体姿态面临巨大挑战:相比专为测量设计的声音源,普通音乐在音量和音高上具有显著波动,这些动态信号变化不可避免地与人体运动引起的声场变化混合,难以提取可靠的姿态线索。为此,BGM2Pose引入对比姿态提取模块,结合对比学习与硬负样本采样,有效剥离录音中的音乐成分,分离出姿态相关信息;同时提出频段注意力模块,通过动态计算各频带注意力,聚焦于由人体运动带来的细微声学变化。实验表明,该方法显著优于现有方法,展现出在真实场景中的巨大应用潜力。相关数据集与代码将公开发布。
原文摘要 · Abstract (English)
We propose BGM2Pose, a non-invasive 3D human pose estimation method using arbitrary music (e.g., background music) as active sensing signals. Unlike existing approaches that significantly limit practicality by employing intrusive chirp signals within the audible range, our method utilizes natural music that causes minimal discomfort to humans. Estimating human poses from standard music presents significant challenges. In contrast to sound sources specifically designed for measurement, regular music varies in both volume and pitch. These dynamic changes in signals caused by music are inevitably mixed with alterations in the sound field resulting from human motion, making it hard to extract reliable cues for pose estimation. To address these challenges, BGM2Pose introduces a Contrastive Pose Extraction Module that employs contrastive learning and hard negative sampling to eliminate musical components from the recorded data, isolating the pose information. Additionally, we propose a Frequency-wise Attention Module that enables the model to focus on subtle acoustic variations attributable to human movement by dynamically computing attention across frequency bands. Experiments suggest that our method outperforms the existing methods, demonstrating substantial potential for real-world applications. Our datasets and code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。