用智能手表音频与运动数据实时识别面对面对话
Detecting In-Person Conversations in Noisy Real-World Environments with Smartwatch Audio and Motion Sensing
- 融合手表麦克风与六轴惯性数据，捕捉对话中的非语言动态
- 实验室和半自然场景下准确率分别达82.0%和77.2%的宏平均F1
- 首次在商用智能手表上实现对话检测的实时部署
社交互动对人类行为、关系与社会结构具有关键影响，涵盖口头交流、非语言手势、面部表情与肢体语言等多种形式。本文提出一种新型计算方法，用于检测面对面口头对话这一核心社交行为。利用市售智能手表同步采集麦克风音频与六轴惯性信号（加速度计与陀螺仪），设计并训练基于卷积与注意力机制的神经网络，采用三种不同融合策略整合音视频与运动模态。通过11名参与者在实验室环境与24名参与者的半自然化研究验证框架有效性。结果表明，融合惯性数据可显著提升检测性能，有效捕捉非语言对话动态。整体框架在实验室环境下实现82.0±3.0%的宏平均F1分数，在半自然化场景中为77.2±1.8%。最后，成功将训练模型部署至商用智能手表上的用户应用，实现对话检测的实时运行。
原文摘要 · Abstract (English)
Social interactions play a crucial role in shaping human behavior, relationships, and societies. It encompasses various forms of communication, such as verbal conversation, non-verbal gestures, facial expressions, and body language. In this work, we develop a novel computational approach to detect face-to-face verbal conversations, a foundational aspect of human social interactions. We leverage multimodal data captured by a commodity smartwatch, specifically synchronizing microphone audio with 6-axis inertial signals (accelerometer and gyroscope). We design, train, and evaluate convolutional and attention-based neural networks using three different fusion methods to integrate the audio and motion modalities. To validate this framework, we conduct a lab study with 11 participants and a semi-naturalistic study with 24 participants. Our comprehensive evaluation demonstrates that fusing inertial data with audio significantly improves detection performance by capturing non-verbal conversational dynamics. Overall, our framework achieved 82.0$\pm$3.0% macro F1-score when detecting conversations in the lab and 77.2$\pm$1.8% in the semi-naturalistic setting. Lastly, we demonstrate real-time conversation detection by deploying our trained model to a user application running on a commercial smartwatch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。