融合视觉与触觉信息,精准估计透明碎片轮廓,助力自动重装。
Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly
- 构建视觉-触觉融合框架,结合摄像头与触觉传感器数据
- 在真实场景中实现95%以上的轮廓匹配准确率
- 适合精密光学仪器修复与文物复原等高价值场景
透明碎片轮廓估计对自主重装至关重要,尤其在精密光学设备维修、文物修复及其他贵重器件破损识别领域。与完整透明物体不同,透明碎片因光学特性严格、形状和边缘不规则,轮廓估计难度更高。本文提出一种基于视觉-触觉融合的通用透明碎片轮廓估计框架。首先,构建名为TransFrag27K的透明碎片数据集,包含多类透明物体在多种场景下的合成数据,以及可扩展的数据生成流程。其次,设计TransFragNet网络以检测、定位并分割采样抓取位置;采用配备Gelsight Mini传感器的双指夹爪获取碎片侧边的触觉重构信息。通过融合触觉与视觉线索,提出视觉-触觉融合材料分类器。受人类通过视触结合推断轮廓的启发,构建了通用透明碎片轮廓估计框架,并在真实世界验证中表现优异。最后,提出基于多维相似性度量的轮廓匹配与重装算法,为视觉-触觉轮廓估计与碎片重装提供可复现的基准。实验结果验证了该框架的有效性。数据集与代码已开源:https://github.com/Keithllin/Transparent-Fragments-Contour-Estimation。
原文摘要 · Abstract (English)
The contour estimation of transparent fragments is very important for autonomous reassembly, especially in the fields of precision optical instrument repair, cultural relic restoration, and identification of other precious device broken accidents. Different from general intact transparent objects, the contour estimation of transparent fragments face greater challenges due to strict optical properties, irregular shapes and edges. To address this issue, a general transparent fragments contour estimation framework based on visual-tactile fusion is proposed in this paper. First, we construct the transparent fragment dataset named TransFrag27K, which includes a multiscene synthetic data of broken fragments from multiple types of transparent objects, and a scalable synthetic data generation pipeline. Secondly, we propose a visual grasping position detection network named TransFragNet to identify, locate and segment the sampling grasping position. And, we use a two-finger gripper with Gelsight Mini sensors to obtain reconstructed tactile information of the lateral edge of the fragments. By fusing this tactile information with visual cues, a visual-tactile fusion material classifier is proposed. Inspired by the way humans estimate a fragment's contour combining vision and touch, we introduce a general transparent fragment contour estimation framework based on visual-tactile fusion, demonstrates strong performance in real-world validation. Finally, a multi-dimensional similarity metrics based contour matching and reassembly algorithm is proposed, providing a reproducible benchmark for evaluating visual-tactile contour estimation and fragment reassembly. The experimental results demonstrate the validity of the proposed framework. The dataset and codes are available at https://github.com/Keithllin/Transparent-Fragments-Contour-Estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。