让AR中虚拟物体碰撞时发出符合材质的实时声音,提升沉浸感。
Sonify Anything: Towards Context-Aware Sonic Interactions in AR
- 通过计算机视觉识别真实物体材质,动态生成匹配的物理声学效果。
- 24人实验显示,材质音效显著提升真实感与材质辨识准确率。
- 适合开发高沉浸式AR应用的团队,尤其注重多感官交互的场景。
在增强现实(AR)中,虚拟物体与真实物体交互时,因缺乏物理质感而常无声音或仅播放通用音效,导致视听不一致,降低交互体验与真实感。与虚拟现实(VR)和游戏不同,AR需实时合成声音以适应新场景和物体,实现音画同步。本文提出一种上下文感知的声音生成框架:利用计算机视觉识别真实物体的材质,结合材质物理属性与碰撞动力学,通过物理建模合成实时音效。24名参与者参与的用户研究发现,基于材质的声音显著提升了交互的真实感,并使用户对视觉相似材质的辨识准确率和信心显著提高。结果表明,上下文感知的材质音效能有效增强AR中的真实感与环境认知。
原文摘要 · Abstract (English)
In Augmented Reality (AR), virtual objects interact with real objects. However, the lack of physicality of virtual objects leads to the absence of natural sonic interactions. When virtual and real objects collide, either no sound or a generic sound is played. Both lead to an incongruent multisensory experience, reducing interaction and object realism. Unlike in Virtual Reality (VR) and games, where predefined scenes and interactions allow for the playback of pre-recorded sound samples, AR requires real-time sound synthesis that dynamically adapts to novel contexts and objects to provide audiovisual congruence during interaction. To enhance real-virtual object interactions in AR, we propose a framework for context-aware sounds using methods from computer vision to recognize and segment the materials of real objects. The material's physical properties and the impact dynamics of the interaction are used to generate material-based sounds in real-time using physical modelling synthesis. In a user study with 24 participants, we compared our congruent material-based sounds to a generic sound effect, mirroring the current standard of non-context-aware sounds in AR applications. The results showed that material-based sounds led to significantly more realistic sonic interactions. Material-based sounds also enabled participants to distinguish visually similar materials with significantly greater accuracy and confidence. These findings show that context-aware, material-based sonic interactions in AR foster a stronger sense of realism and enhance our perception of real-world surroundings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。