OpenGlass让眼镜本地运行大模型,实时帮视障者看懂周围环境。
OpenGlass: A Sensing-Computing Split Architecture for Local MLLM-Driven Real-Time Visual Assistance

- 眼镜端采集视觉数据,手机等设备本地运行大模型处理。
- 993毫秒内完成从看到语音输出,97.5%响应在2秒内。
- 开源隐私保护,适合开发无障碍视觉辅助工具的人看。
我们提出OpenGlass,一个开源、注重隐私、以本地优先的低延迟多模态视觉辅助系统,主要面向盲人和低视力用户。云端多模态大模型(MLLM)虽具备强大视觉理解能力,但通常需上传第一人称视觉数据,并存在数秒级网络延迟;可穿戴眼镜适合感知,却受限于算力与功耗无法运行大型模型。OpenGlass通过感知-计算分离架构解决此问题:基于ESP32的眼镜端负责采集视觉上下文,附近消费级设备执行本地MLLM推理与语音输出,减少对云服务依赖,并默认将原始第一人称视觉数据保留在用户可控设备上。我们评估了响应质量、查询到音频的延迟、安全敏感时的拒绝响应及可审计日志。在真实ESP32 Wi-Fi捕获条件下,使用缩放后数据包的中位用户到音频延迟为993毫秒,使用原始1280×720数据包为1625毫秒;分别有97.5%和93.3%的试验响应低于2秒。OpenGlass是用户主动触发的视觉辅助参考平台,适用于障碍物/危险识别、标识/物体查询及图像质量自检,而非认证导航工具。我们已公开源代码、硬件说明、提示词、评估数据与日志。
原文摘要 · Abstract (English)
We present OpenGlass, an open-source, privacy-oriented, local-first system for low-latency multimodal visual assistance, with a primary focus on blind and low-vision users. Cloud MLLM assistants offer strong visual understanding, but often require uploading first-person visual data and can suffer multi-second network delays; wearable glasses are ideal for sensing, but cannot host large models under tight compute and power budgets. OpenGlass addresses this gap with a sensing-computing split: an ESP32-based glasses-side unit captures visual context, while a nearby consumer-grade device performs local MLLM inference and local speech output, reducing cloud reliance and keeping raw egocentric visual data on user-controlled devices by default. We evaluate response quality, query-ready-to-audio latency, safety-aware abstention, and auditable logs. Under real ESP32 Wi-Fi capture, OpenGlass reaches 993 ms median user-to-audio latency with resized payloads and 1625 ms with raw 1280 x 720 payloads; 97.5% and 93.3% of trials fall below 2 s, respectively. OpenGlass is a user-initiated visual-assistance reference platform for obstacle/hazard awareness, sign/object queries, and image-quality self-checking, rather than a certified navigation aid. We release source code, hardware instructions, prompts, evaluation data, and logs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。