arXiv:2409.13926cs.AIcs.HC2024-09被引 20

用生成AI把真实环境融入虚拟空间,提升远程协作沉浸感。

SpaceBlender: Creating Context-Rich Collaborative Spaces Through Generative 3D Scene Blending

  • 通过深度估计与扩散模型,将用户实拍照片融合成3D虚拟空间。
  • 20人实验显示,混合空间显著提升熟悉感和协作体验。
  • 适合远程会议、协同设计等需真实情境的VR场景。

当前生成式AI在虚拟现实(VR)应用中生成的3D空间多为人工构造,难以支持需要结合用户真实环境的协作任务。为此,我们提出SpaceBlender,一种新颖的流水线,利用生成式AI将用户的物理环境融合进统一的虚拟空间中。该方法通过迭代过程:深度估计、网格对齐及基于几何先验与自适应文本提示的扩散式空间补全,将用户提供的2D图像转化为富含上下文的3D环境。在一项包含20名参与者的配对对照研究中,参与者完成协作式的亲缘图绘制任务,比较了SpaceBlender与通用虚拟环境及现有先进场景生成框架的表现。结果显示,参与者普遍认为SpaceBlender提供的环境更具熟悉感和情境相关性,但也指出生成空间中的复杂性可能分散任务注意力。基于反馈,我们提出了改进方向,并讨论了混合空间在不同场景下的价值与设计策略。

原文摘要 · Abstract (English)

There is increased interest in using generative AI to create 3D spaces for Virtual Reality (VR) applications. However, today's models produce artificial environments, falling short of supporting collaborative tasks that benefit from incorporating the user's physical context. To generate environments that support VR telepresence, we introduce SpaceBlender, a novel pipeline that utilizes generative AI techniques to blend users' physical surroundings into unified virtual spaces. This pipeline transforms user-provided 2D images into context-rich 3D environments through an iterative process consisting of depth estimation, mesh alignment, and diffusion-based space completion guided by geometric priors and adaptive text prompts. In a preliminary within-subjects study, where 20 participants performed a collaborative VR affinity diagramming task in pairs, we compared SpaceBlender with a generic virtual environment and a state-of-the-art scene generation framework, evaluating its ability to create virtual spaces suitable for collaboration. Participants appreciated the enhanced familiarity and context provided by SpaceBlender but also noted complexities in the generative environments that could detract from task focus. Drawing on participant feedback, we propose directions for improving the pipeline and discuss the value and design of blended spaces for different scenarios.

虚拟现实生成式AI空间融合协作交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。