评测大模型在XR设备上的表现，帮开发者选对模型和设备。
AIvaluateXR: An Evaluation Framework for on-Device AI in XR with Benchmarking Results
- 构建3D帕累托最优框架，综合评估性能与速度
- 测试68组模型-设备组合，覆盖4大平台与多种参数
- 对比本地、云端部署效果，指导实时交互优化
大型语言模型（LLMs）在扩展现实（XR）设备上的部署极大推动了人机交互发展。然而，在直接本地推理场景下，针对特定任务选择合适的模型与设备仍具挑战。本文提出AIvaluateXR，一个用于评估运行于XR设备上LLMs的综合性基准框架。为验证该框架，我们在Magic Leap 2、Meta Quest 3、Vivo X100s Pro和Apple Vision Pro四个平台上部署了17个选定的LLM，并进行广泛评估。实验测量四项关键指标：性能一致性、处理速度、内存占用与电池消耗。针对每个68组模型-设备组合，分析不同输入长度、批处理大小和线程数下的表现，权衡实时XR应用中的取舍。我们基于3D帕累托最优理论提出统一评估方法，以从质量和速度双重目标中筛选最优模型-设备配对。此外，还比较了本地、客户端-服务器与云端部署的效率，并评估其在两项交互任务中的准确性。研究结果为未来在XR设备上部署LLM的优化提供重要参考。该评估方法可作为该新兴领域研究与开发的标准基础。源代码与补充材料见：www.nanovis.org/AIvaluateXR.html
原文摘要 · Abstract (English)
The deployment of large language models (LLMs) on extended reality (XR) devices has great potential to advance the field of human-AI interaction. In the case of direct, on-device model inference, selecting the appropriate model and device for specific tasks remains challenging. In this paper, we present AIvaluateXR, a comprehensive evaluation framework for benchmarking LLMs running on XR devices. To demonstrate the framework, we deploy 17 selected LLMs across four XR platforms: Magic Leap 2, Meta Quest 3, Vivo X100s Pro, and Apple Vision Pro, and conduct an extensive evaluation. Our experimental setup measures four key metrics: performance consistency, processing speed, memory usage, and battery consumption. For each of the 68 model-device pairs, we assess performance under varying string lengths, batch sizes, and thread counts, analyzing the trade-offs for real-time XR applications. We propose a unified evaluation method based on the 3D Pareto Optimality theory to select the optimal device-model pairs from quality and speed objectives. Additionally, we compare the efficiency of on-device LLMs with client-server and cloud-based setups, and evaluate their accuracy on two interactive tasks. We believe our findings offer valuable insight to guide future optimization efforts for LLM deployment on XR devices. Our evaluation method can be used as standard groundwork for further research and development in this emerging field. The source code and supplementary materials are available at: www.nanovis.org/AIvaluateXR.html
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。