开源工具包CUIfy让虚拟现实中的角色会说话会对话。
CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR
- 用大模型+语音识别合成技术，让虚拟角色能实时对话
- 支持多个角色同时互动，端到端延迟低至可交互水平
- 开源可定制，适合游戏、培训等需要自然交互的场景
计算机图形学、机器学习和传感器技术的进步为扩展现实（XR）在日常生活的应用创造了诸多机会，涵盖技能培训到娱乐。随着大型企业推出价格合理的消费级头戴显示设备（HMD），XR有望普及，未来将如智能手机和平板一样成为个人设备。然而，实现智能空间与自然交互同样关键，以提升用户在虚拟和增强空间中的参与度。为此，采用大语言模型（LLM）驱动的非玩家角色（NPC），结合语音转文字（STT）和文字转语音（TTS）模型，相较于传统预设脚本的NPC，在构建自然对话式用户界面（CUI）方面具有显著优势。本文发布了一个开源、可定制、可扩展且注重隐私保护的Unity工具包CUIfy，支持通过主流LLM、STT和TTS模型实现基于语音的NPC-用户交互。该工具包支持环境中多个LLM驱动的NPC共存，并通过流式处理降低不同计算模型间的延迟，从而实现用户与NPC之间可用的实时交互。源代码已公开于：https://gitlab.lrz.de/hctl/cuify
原文摘要 · Abstract (English)
Recent developments in computer graphics, machine learning, and sensor technologies enable numerous opportunities for extended reality (XR) setups for everyday life, from skills training to entertainment. With large corporations offering affordable consumer-grade head-mounted displays (HMDs), XR will likely become pervasive, and HMDs will develop as personal devices like smartphones and tablets. However, having intelligent spaces and naturalistic interactions in XR is as important as technological advances so that users grow their engagement in virtual and augmented spaces. To this end, large language model (LLM)--powered non-player characters (NPCs) with speech-to-text (STT) and text-to-speech (TTS) models bring significant advantages over conventional or pre-scripted NPCs for facilitating more natural conversational user interfaces (CUIs) in XR. This paper provides the community with an open-source, customizable, extendable, and privacy-aware Unity package, CUIfy, that facilitates speech-based NPC-user interaction with widely used LLMs, STT, and TTS models. Our package also supports multiple LLM-powered NPCs per environment and minimizes latency between different computational models through streaming to achieve usable interactions between users and NPCs. We publish our source code in the following repository: https://gitlab.lrz.de/hctl/cuify
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。