用手套融合视觉与声音,让残障者更轻松抓取透明物品。
MultiClear: Multimodal Soft Exoskeleton Glove for Transparent Object Grasping Assistance
- 用多模态数据融合提升软体手套的抓握感知能力。
- 在透明物体抓取任务中达到70.37%的成功率。
- 适合残障辅助、智能穿戴与人机交互研究者参考。
抓取是与环境互动的基础能力,但对部分人群(如残障者)而言极具挑战。可穿戴机器人系统可增强或恢复手部功能,近年计算机视觉的进步提升了抓取能力,但透明物体因视觉对比度低、深度线索模糊,仍难处理。尽管已有结合触觉与听觉反馈的多模态策略,但视觉与这些模态的融合仍不充分。本文提出MultiClear,一种用于可穿戴软体外骨骼手套的多模态框架,通过融合RGB、深度数据和音频信号,提升透明物体的抓取辅助能力。手套集成肌腱驱动执行器、RGB-D相机与内置麦克风。采用分层控制架构:高层提供上下文感知,中层处理多模态输入,底层执行PID电机控制以实现精细调整。透明物体分割问题通过引入零样本分割的视觉基础模型解决。实验表明,该系统在透明物体操作中取得70.37%的抓取能力得分,验证了其有效性。
原文摘要 · Abstract (English)
Grasping is a fundamental skill for interacting with the environment. However, this ability can be difficult for some (e.g. due to disability). Wearable robotic solutions can enhance or restore hand function, and recent advances have leveraged computer vision to improve grasping capabilities. However, grasping transparent objects remains challenging due to their poor visual contrast and ambiguous depth cues. Furthermore, while multimodal control strategies incorporating tactile and auditory feedback have been explored to grasp transparent objects, the integration of vision with these modalities remains underdeveloped. This paper introduces MultiClear, a multimodal framework designed to enhance grasping assistance in a wearable soft exoskeleton glove for transparent objects by fusing RGB data, depth data, and auditory signals. The exoskeleton glove integrates a tendon-driven actuator with an RGB-D camera and a built-in microphone. To achieve precise and adaptive control, a hierarchical control architecture is proposed. For the proposed hierarchical control architecture, a high-level control layer provides contextual awareness, a mid-level control layer processes multimodal sensory inputs, and a low-level control executes PID motor control for fine-tuned grasping adjustments. The challenge of transparent object segmentation was managed by introducing a vision foundation model for zero-shot segmentation. The proposed system achieves a Grasping Ability Score of 70.37%, demonstrating its effectiveness in transparent object manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。