用大模型生成JSON数据，让XR世界更智能地响应自然语言。
LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models
- 将自然语言转为结构化JSON数据，避免脚本错误导致崩溃。
- 用户任务完成时间减少60%，令牌消耗降低80%以上。
- 适合开发交互式虚拟现实应用的团队快速构建场景。
大型语言模型（LLM）与扩展现实（XR）技术的融合，有望实现通过自然语言与用户互动的沉浸式XR环境，例如从语音输入生成并动画化3D场景。然而，XR环境的复杂性使得从海量XR产物中准确提取上下文数据和场景/对象参数变得困难，导致使用按需付费模型时成本上升，且生成错误率升高。现有聚焦代码脚本生成的方法常出现错误，引发脚本无效、应用崩溃，最终降低用户体验。为此，我们提出LLMER框架，利用大模型生成的JSON数据创建可交互的XR世界。不同于以往关注代码生成的方法，LLMER将自然语言输入转化为JSON数据，显著降低应用崩溃风险与处理延迟。该框架采用多阶段策略，仅提供用户请求所需的必要上下文信息，并配备多个模块以应对不同XR任务。初步用户研究表明，系统在令牌消耗上减少超80%，任务完成时间缩短约60%。用户反馈分析也揭示了进一步优化的方向。
原文摘要 · Abstract (English)
The integration of Large Language Models (LLMs) like GPT-4 with Extended Reality (XR) technologies offers the potential to build truly immersive XR environments that interact with human users through natural language, e.g., generating and animating 3D scenes from audio inputs. However, the complexity of XR environments makes it difficult to accurately extract relevant contextual data and scene/object parameters from an overwhelming volume of XR artifacts. It leads to not only increased costs with pay-per-use models, but also elevated levels of generation errors. Moreover, existing approaches focusing on coding script generation are often prone to generation errors, resulting in flawed or invalid scripts, application crashes, and ultimately a degraded user experience. To overcome these challenges, we introduce LLMER, a novel framework that creates interactive XR worlds using JSON data generated by LLMs. Unlike prior approaches focusing on coding script generation, LLMER translates natural language inputs into JSON data, significantly reducing the likelihood of application crashes and processing latency. It employs a multi-stage strategy to supply only the essential contextual information adapted to the user's request and features multiple modules designed for various XR tasks. Our preliminary user study reveals the effectiveness of the proposed system, with over 80% reduction in consumed tokens and around 60% reduction in task completion time compared to state-of-the-art approaches. The analysis of users' feedback also illuminates a series of directions for further optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。