arXiv:2411.16802cs.ROcs.AI2024-11

用大模型指导小模型,精准定位熔池变形体关键点。

Leveraging Foundation Models To learn the shape of semi-fluid deformable objects

  • 用大模型作教师,无需预训练直接提取图像像素级特征
  • 学生模型定位关键点误差仅13.4像素
  • 适合机器人操控半流体物体的场景

操控半流体可变形物体的一大挑战在于其形态表征与关键点检测。尽管过去十年研究聚焦于非流体类物体(如衣物、绳索)的表征与操作,但通常仍需依赖图像中的像素级信息,并通过人工标注数据训练分割网络来获取。本文聚焦于熔池形态表征,旨在定义可用于后续运动控制的稳定特征。提出两种方法:第一种采用教师-学生框架训练生成模型以表征流体可变形物体;第二种则直接利用基础模型作为教师,无需预训练或特定数据集即可完成图像中物体的表征。知识蒸馏结果表明,学生网络能以13.4像素误差准确恢复物体关键点;教师模型在像素级掩码重建上达到75.26%的平均交并比(mIoU)。

原文摘要 · Abstract (English)

One of the difficulties imposed on the manipulation of deformable objects is their characterization and the detection of representative keypoints for the purpose of manipulation. A keen interest was manifested by researchers in the last decade to characterize and manipulate deformable objects of non-fluid nature, such as clothes and ropes. Even though several propositions were made in the regard of object characterization, however researchers were always confronted with the need of pixel-level information of the object through images to extract relevant information. This usually is accomplished by means of segmentation networks trained on manually labeled data for this purpose. In this paper, we address the subject of characterizing weld pool to define stable features that serve as information for further motion control objectives. We achieve this by employing different pipelines. The first one consists of characterizing fluid deformable objects through the use of a generative model that is trained using a teacher-student framework. And in the second one we leverage foundation models by using them as teachers to characterize the object in the image, without the need of any pre-training and any dataset. The performance of knowledge distillation from foundation models into a smaller generative model shows prominent results in the characterization of deformable objects. The student network was capable of learning to retrieve the keypoitns of the object with an error of 13.4 pixels. And the teacher was evaluated based on its capacities to retrieve pixel level information represented by the object mask, with a mean Intersection Over Union (mIoU) of 75.26%.

可变形物体关键点检测知识蒸馏机器人操控

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。