用手机实现离线视觉辅助,能找个人物品并实时导航。
VisionAId: An Offline-First Multimodal Android Assistant for People with Visual Impairment, Featuring Personalized Object Retrieval

- 手机本地运行6个AI模型,离线工作不依赖云端。
- 自拍几秒就能定位特定物品,误差小于1厘米。
- 适合视障者日常使用,支持语音、震动多模态反馈。
全球超过2.85亿人患有视觉障碍,日常如避障、找物品、认人脸或辨钞票都难以自理。现有助盲应用通常仅识别预设类别,严重依赖云服务或专用硬件。我们提出VisionAId,一款Android应用,将普通手机变为实时视觉助手。系统集成六个本地运行的深度学习模型(单目测距、实例分割、视觉与人脸嵌入、人脸检测及定制钞票检测器),全部通过ONNX Runtime执行;仅在生成场景描述和自动标注时可选调用云端大模型(Google Gemini Flash)。其核心创新是少样本物体检索流程:用户拍摄物品多角度照片后,系统可后续在环境中定位该具体物品,通过增强现实标记、空间音频及距离感知振动引导用户。所有反馈采用多模态形式(罗马尼亚语语音合成、语音指令、震动)。在参考设备(三星Galaxy S21 Ultra)上,INT8量化使深度推理延迟从约1200毫秒降至约491毫秒,定制钞票检测器达到mAP@50为0.986,且3米内测距误差低于1厘米。
原文摘要 · Abstract (English)
Over 285 million people worldwide live with a visual impairment, for whom everyday tasks such as avoiding obstacles, locating personal belongings, recognizing familiar faces, or handling cash remain persistent obstacles to personal autonomy. Existing assistive applications are typically limited to recognizing predefined categories, depend heavily on cloud connectivity, or require dedicated hardware. We present VisionAId, an Android application that turns a commodity smartphone into a real-time visual assistant. The system integrates six on-device deep learning models (metric monocular depth estimation, instance segmentation, visual and facial embeddings, face detection, and a custom banknote detector) running entirely through ONNX Runtime, with an optional cloud large language model (Google Gemini Flash) used only for narrative scene description and automatic object labeling. A distinctive contribution is a few-shot pipeline for personal objects: the user photographs an object from several angles, and the system later locates that specific instance in the environment, guiding the user toward it with augmented-reality markers, spatial audio, and distance-proportional haptics. All feedback is multimodal (Romanian speech synthesis, voice commands, vibration). On a reference device (Samsung Galaxy S21 Ultra), INT8 quantization reduces depth latency from ~1200 ms to ~491 ms, the custom banknote detector reaches an mAP@50 of 0.986, and metric depth is calibrated to below 1 cm of error within 3 m.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。