用视觉算法帮视障学生听懂机器人路径，玩转编程与机器人
Inclusive STEAM Education: A Framework for Teaching Cod-2 ing and Robotics to Students with Visually Impairment Using 3 Advanced Computer Vision
- 用CLIP把摄像头拍的迷宫图转成语音提示，让视障生听懂空间布局
- 机器人配双目摄像头+SLAM实时传位置，语音指令经CLIP优化更精准
- 适合特殊教育、无障碍科技研发者，推动视障人群参与STEAM
STEAM教育融合科学、技术、工程、艺术与数学以培养创造力与问题解决能力。然而，视障学生在编程与机器人学习中面临追踪机器人运动和建立空间意识的重大挑战。本文提出一个框架，利用预构建机器人与算法（如迷宫求解技术），在可访问的学习环境中实现教学。系统采用对比语言-图像预训练（CLIP）处理全局摄像头捕捉的迷宫布局，将视觉数据转化为文本描述，并生成空间音频提示至音频虚拟现实（AVR）系统。学生发出语音命令，经由CLIP优化；同时，安装在机器人上的立体摄像头提供实时数据，通过同步定位与地图构建（SLAM）实现持续反馈。该框架使视障学生能发展编程技能并参与复杂问题解决任务。除迷宫求解外，该方法还展示了计算机视觉在特殊教育中的广泛潜力，有助于提升STEAM领域教育的可及性与学习体验。
原文摘要 · Abstract (English)
STEAM education integrates Science, Technology, Engineering, Arts, and Mathematics to foster creativity and problem-solving. However, students with visual impairments (VI) encounter significant challenges in programming and robotics, particularly in tracking robot movements and developing spatial awareness. This paper presents a framework that leverages pre-constructed robots and algorithms, such as maze-solving techniques, within an accessible learning environment. The proposed system employs Contrastive Language-Image Pre-training (CLIP) to process global camera-captured maze layouts, converting visual data into textual descriptions that generate spatial audio prompts in an Audio Virtual Reality (AVR) system. Students issue verbal commands, which are refined through CLIP, while robot-mounted stereo cameras provide real-time data processed via Simultaneous Localization and Mapping (SLAM) for continuous feedback. By integrating these technologies, the framework empowers VI students to develop coding skills and engage in complex problem-solving tasks. Beyond maze-solving applications, this approach demonstrates the broader potential of computer vision in special education, contributing to improved accessibility and learning experiences in STEAM disciplines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。