用手机摄像头+树莓派实现室内盲人导航，实时语音指引路线
Turn-by-Turn Indoor Navigation for the Visually Impaired
- 手机拍图传给树莓派，本地运行多模态模型识别环境
- 通过本地LLM将视觉信息转为自然语言指令，准确率高
- 无需云端、不依赖设备算力，适合所有手机，隐私安全
由于复杂布局和缺乏GPS信号，视障人士在室内环境中导航面临重大挑战。本文提出一种新系统，仅使用配备摄像头的智能手机，结合多模态模型、深度学习算法和大语言模型（LLMs），实现建筑物内的逐步导航。智能手机摄像头实时捕捉周围图像，传输至附近的树莓派设备，该设备可本地运行LLM、多模态模型及深度学习算法，用于检测和识别建筑特征、标识和障碍物。解析后的视觉数据由运行在树莓派上的LLM转化为自然语言指令，并回传至用户，通过语音提示提供直观且上下文感知的引导。该方案大幅降低用户设备负担，避免设备过载，兼容所有类型设备，包括无法运行AI模型的机型。同时，确保训练数据和其他信息不离开建筑，提升隐私安全性。初步评估表明，该系统在复杂室内空间中能有效精准引导用户，具备广泛应用潜力。
原文摘要 · Abstract (English)
Navigating indoor environments presents significant challenges for visually impaired individuals due to complex layouts and the absence of GPS signals. This paper introduces a novel system that provides turn-by-turn navigation inside buildings using only a smartphone equipped with a camera, leveraging multimodal models, deep learning algorithms, and large language models (LLMs). The smartphone's camera captures real-time images of the surroundings, which are then sent to a nearby Raspberry Pi capable of running on-device LLM models, multimodal models, and deep learning algorithms to detect and recognize architectural features, signage, and obstacles. The interpreted visual data is then translated into natural language instructions by an LLM running on the Raspberry Pi, which is sent back to the user, offering intuitive and context-aware guidance via audio prompts. This solution requires minimal workload on the user's device, preventing it from being overloaded and offering compatibility with all types of devices, including those incapable of running AI models. This approach enables the client to not only run advanced models but also ensure that the training data and other information do not leave the building. Preliminary evaluations demonstrate the system's effectiveness in accurately guiding users through complex indoor spaces, highlighting its potential for widespread application
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。