用生成式AI让虚拟现实交互更自然，内容自动生成降低成本
When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions
- 用语言指令驱动3D内容生成与操作，替代复杂编程
- 通过视觉-语言模型理解场景，实现自然交互与内容自动构建
- 适合教育、培训等需大规模沉浸式内容的场景
扩展现实（XR）涵盖虚拟现实、增强现实和混合现实，广泛应用于教育、辅助和训练等领域。然而，其普及受限于两大挑战：一是大规模或复杂交互场景的3D内容制作成本高、流程复杂；二是非直观交互方式（如手持控制器或预设手势）学习门槛高。生成式AI（GenAI）通过语言驱动交互和自动化内容生成，提供突破性解决方案。借助视觉-语言模型与基于扩散的生成技术，GenAI可理解模糊指令、解析物理场景并生成或修改3D内容，显著降低使用门槛。本文通过三个具体应用场景，验证了该融合在提升可扩展性与自然交互方面的潜力，并识别出亟待解决的技术瓶颈以推动更广泛应用。
原文摘要 · Abstract (English)
Extended Reality (XR), including virtual, augmented, and mixed reality, provides immersive and interactive experiences across diverse applications, from VR-based education to AR-based assistance and MR-based training. However, widespread XR adoption remains limited due to two key challenges: 1) the high cost and complexity of authoring 3D content, especially for large-scale environments or complex interactions; and 2) the steep learning curve associated with non-intuitive interaction methods like handheld controllers or scripted gestures. Generative AI (GenAI) presents a promising solution by enabling intuitive, language-driven interaction and automating content generation. Leveraging vision-language models and diffusion-based generation, GenAI can interpret ambiguous instructions, understand physical scenes, and generate or manipulate 3D content, significantly lowering barriers to XR adoption. This paper explores the integration of XR and GenAI through three concrete use cases, showing how they address key obstacles in scalability and natural interaction, and identifying technical challenges that must be resolved to enable broader adoption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。