基于视觉语言模型的实时随身智能助手，可持续观察环境并提供自然对话与任务指导。
Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model
- 通过头戴视角多模态模型实现持续感知与实时交互
- 支持当前观察与历史上下文问答，并能规划后续任务
- 集成视频生成模块，为复杂任务生成分步演示
我们提出 Vinci，一个基于头戴视角视觉语言模型的实时具身智能助手。该系统专为手机和可穿戴摄像头等便携设备设计，采用“始终开启”模式，持续观察环境以实现无缝交互与辅助。用户可通过唤醒系统进行自然语言对话，提问或寻求帮助，系统以语音回应，支持无手操作。Vinci 能实时处理长视频流，回答关于当前观测与历史背景的问题，并基于过往互动进行任务规划。为进一步提升可用性，系统集成视频生成模块，可生成任务所需的分步视觉演示。我们希望 Vinci 能建立便携式、实时头戴式 AI 系统的坚实框架，为用户提供情境化且可操作的洞察。完整实现代码及演示平台已开源，可通过 https://github.com/OpenGVLab/vinci 测试上传视频。
原文摘要 · Abstract (English)
We introduce Vinci, a real-time embodied smart assistant built upon an egocentric vision-language model. Designed for deployment on portable devices such as smartphones and wearable cameras, Vinci operates in an "always on" mode, continuously observing the environment to deliver seamless interaction and assistance. Users can wake up the system and engage in natural conversations to ask questions or seek assistance, with responses delivered through audio for hands-free convenience. With its ability to process long video streams in real-time, Vinci can answer user queries about current observations and historical context while also providing task planning based on past interactions. To further enhance usability, Vinci integrates a video generation module that creates step-by-step visual demonstrations for tasks that require detailed guidance. We hope that Vinci can establish a robust framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. We release the complete implementation for the development of the device in conjunction with a demo web platform to test uploaded videos at https://github.com/OpenGVLab/vinci.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。