arXiv:2508.06566cs.CVcs.AI2025-08被引 6

用触觉与视觉融合数据,实现高精度低延迟的表面材质识别。

Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Features

  • 基于Transformer架构,融合结构化触觉特征与降维后视觉嵌入
  • 触觉仅输入时达99.4%准确率,推理仅0.77毫秒
  • 适合需实时响应的机器人感知任务

表面材质识别是机器人感知与物理交互的关键环节,尤其在融合触觉与视觉信息时。本文提出Surformer v1,一种基于Transformer的架构,利用结构化触觉特征和通过ResNet-50提取并经主成分分析(PCA)降维的视觉嵌入进行分类。模型采用模态专用编码器与跨模态注意力层,实现视觉与触觉间的丰富交互。首先,仅使用触觉数据的实验中,我们通过特征工程训练并评估多个机器学习模型,发现专为触觉设计的编码器仅Transformer模型在准确率与推理速度上均表现最佳。随后,在多模态融合设置下,对比了使用结构化特征的Surformer v1与使用原始图像的多模态CNN。结果表明,Surformer v1在99.4%准确率下仅需0.77毫秒推理时间,而多模态CNN虽准确率略高但耗时显著增加。这表明Surformer v1在准确性、效率与计算成本之间实现了优异平衡,适用于表面材质识别任务。

原文摘要 · Abstract (English)

Surface material recognition is a key component in robotic perception and physical interaction, particularly when leveraging both tactile and visual sensory inputs. In this work, we propose Surformer v1, a transformer-based architecture designed for surface classification using structured tactile features and PCA-reduced visual embeddings extracted via ResNet-50. The model integrates modality-specific encoders with cross-modal attention layers, enabling rich interactions between vision and touch. Currently, state-of-the-art deep learning models for vision tasks have achieved remarkable performance. With this in mind, our first set of experiments focused exclusively on tactile-only surface classification. Using feature engineering, we trained and evaluated multiple machine learning models, assessing their accuracy and inference time. We then implemented an encoder-only Transformer model tailored for tactile features. This model not only achieved the highest accuracy but also demonstrated significantly faster inference time compared to other evaluated models, highlighting its potential for real-time applications. To extend this investigation, we introduced a multimodal fusion setup by combining vision and tactile inputs. We trained both Surformer v1 (using structured features) and Multimodal CNN (using raw images) to examine the impact of feature-based versus image-based multimodal learning on classification accuracy and computational efficiency. The results showed that Surformer v1 achieved 99.4% accuracy with an inference time of 0.77 ms, while the Multimodal CNN achieved slightly higher accuracy but required significantly more inference time. These findings suggest Surformer v1 offers a compelling balance between accuracy, efficiency, and computational cost for surface material recognition.

表面识别多模态融合Transformer机器人感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。