开发可触觉与视觉融合的智能指尖传感器，提升机器人抓取感知能力。
Design and Benchmarking of A Multi-Modality Sensor for Robotic Manipulation with GAN-Based Cross-Modality Interpretation
- 设计透明皮肤与仿生触点，实现触觉与视觉协同感知。
- 通过多任务学习模型同步识别硬度、材质和纹理特征。
- 基于GAN实现跨模态理解，适配复杂环境下的机器人操作。
本文提出一种新型多模态传感器ViTacTip，满足紧凑设计下先进多模态传感需求。其透明皮肤采用'透视皮肤'机制，可在接触时捕捉物体细节，显著提升视觉与近距感知能力；嵌入式仿生触点则放大接触信息，大幅提升触觉及衍生力觉感知。为验证多模态性能，我们构建了多任务学习模型，可同时识别硬度、材料与纹理。通过广泛基准测试，涵盖物体识别、接触点检测、姿态回归与条纹识别等任务。为实现模态间无缝切换，采用生成对抗网络（GAN）方法，实现跨模态解释，增强传感器在多样化环境中的适用性。
原文摘要 · Abstract (English)
In this paper, we present the design and benchmark of an innovative sensor, ViTacTip, which fulfills the demand for advanced multi-modal sensing in a compact design. A notable feature of ViTacTip is its transparent skin, which incorporates a `see-through-skin' mechanism. This mechanism aims at capturing detailed object features upon contact, significantly improving both vision-based and proximity perception capabilities. In parallel, the biomimetic tips embedded in the sensor's skin are designed to amplify contact details, thus substantially augmenting tactile and derived force perception abilities. To demonstrate the multi-modal capabilities of ViTacTip, we developed a multi-task learning model that enables simultaneous recognition of hardness, material, and textures. To assess the functionality and validate the versatility of ViTacTip, we conducted extensive benchmarking experiments, including object recognition, contact point detection, pose regression, and grating identification. To facilitate seamless switching between various sensing modalities, we employed a Generative Adversarial Network (GAN)-based approach. This method enhances the applicability of the ViTacTip sensor across diverse environments by enabling cross-modality interpretation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。