在扩展现实里用手柄点击实物，本地AI即时回答问题。
ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality
- 用控制器点击现实物体，本地运行视觉语言模型响应
- 用户研究显示延迟适中，体验可接受，隐私更安全
- 适合注重隐私和实时性的沉浸式交互应用
我们提出 ClickAIXR，一种在扩展现实（XR）中实现与真实物体进行多模态视觉-语言交互的本地化框架。不同于依赖云端AI（如ChatGPT）或注视选择（如GazePointAR）的系统，ClickAIXR 将本地运行的视觉语言模型（VLM）与控制器操作结合，使用户能够精确点击现实中的物体。选中后，物体图像由本地VLM处理，通过文本和语音回答自然语言问题。这种以物体为中心的交互方式减少了仅靠注视或语音带来的歧义，且所有推理均在设备端完成，提升了透明性并缓解了隐私与延迟顾虑。我们在 Magic Leap SDK（C API）中实现了该系统，采用 ONNX 本地部署 VLM。用户研究表明，与 Gemini 2.5 Flash 及 ChatGPT 5 相比，其延迟适中，用户体验可接受。结果表明，基于点击选择与本地AI的交互方式，有望推动可信、隐私保护的XR人机交互发展。源码与补充材料见：nanovis.org/ClickAIXR.html
原文摘要 · Abstract (English)
We present ClickAIXR, a novel on-device framework for multimodal vision-language interaction with objects in extended reality (XR). Unlike prior systems that rely on cloud-based AI (e.g., ChatGPT) or gaze-based selection (e.g., GazePointAR), ClickAIXR integrates an on-device vision-language model (VLM) with a controller-based object selection paradigm, enabling users to precisely click on real-world objects in XR. Once selected, the object image is processed locally by the VLM to answer natural language questions through both text and speech. This object-centered interaction reduces ambiguity inherent in gaze- or voice-only interfaces and improves transparency by performing all inference on-device, addressing concerns around privacy and latency. We implemented ClickAIXR in the Magic Leap SDK (C API) with ONNX-based local VLM inference. We conducted a user study comparing ClickAIXR with Gemini 2.5 Flash and ChatGPT 5, evaluating usability, trust, and user satisfaction. Results show that latency is moderate and user experience is acceptable. Our findings demonstrate the potential of click-based object selection combined with on-device AI to advance trustworthy, privacy-preserving XR interactions. The source code and supplementary materials are available at: nanovis.org/ClickAIXR.html
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。