基于视觉语言模型的便携式实时智能助手,支持多场景即时辅助。
An Egocentric Vision-Language Model based Portable Real-time Smart Assistant
- 融合第一人称视觉与大语言模型,实现场景理解与动作规划。
- 在多个公开数据集上表现优异,支持长视频实时处理与上下文记忆。
- 无需专用硬件,适配手机、可穿戴设备,适合日常使用场景。
我们提出Vinci,一个面向便携设备的实时综合智能助手系统。核心是EgoVideo-VL模型,将第一人称视觉基础模型与大语言模型结合,实现场景理解、时间定位、视频摘要和未来规划等高级功能。为提升实用性,Vinci引入记忆模块以实现实时处理长视频并保留上下文历史,生成模块用于输出视觉动作示范,检索模块则连接第一人称与第三人称视角,提供技能学习所需的参考视频。与依赖专用硬件的系统不同,Vinci具备硬件无关性,可在智能手机、可穿戴相机等多种设备上部署。实验表明,EgoVideo-VL在多个公开基准上表现优越,展现了强大的视觉-语言推理与上下文理解能力。用户研究进一步验证了Vinci在多样化真实场景中的适应性与可用性。完整代码(含前端、后端及模型)已开源至https://github.com/OpenGVLab/vinci,旨在建立便携式实时第一人称AI系统的新型范式,为用户提供情境化且可操作的洞察。
原文摘要 · Abstract (English)
We present Vinci, a vision-language system designed to provide real-time, comprehensive AI assistance on portable devices. At its core, Vinci leverages EgoVideo-VL, a novel model that integrates an egocentric vision foundation model with a large language model (LLM), enabling advanced functionalities such as scene understanding, temporal grounding, video summarization, and future planning. To enhance its utility, Vinci incorporates a memory module for processing long video streams in real time while retaining contextual history, a generation module for producing visual action demonstrations, and a retrieval module that bridges egocentric and third-person perspectives to provide relevant how-to videos for skill acquisition. Unlike existing systems that often depend on specialized hardware, Vinci is hardware-agnostic, supporting deployment across a wide range of devices, including smartphones and wearable cameras. In our experiments, we first demonstrate the superior performance of EgoVideo-VL on multiple public benchmarks, showcasing its vision-language reasoning and contextual understanding capabilities. We then conduct a series of user studies to evaluate the real-world effectiveness of Vinci, highlighting its adaptability and usability in diverse scenarios. We hope Vinci can establish a new framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. Including the frontend, backend, and models, all codes of Vinci are available at https://github.com/OpenGVLab/vinci.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。