在极稀疏视觉下实现表面材质识别与重建,验证了预训练模型的潜力。
Seeing Through Extreme Visual Sparsity: Surface Understanding from a Single Random Visual Patch

- 设计统一双任务框架,融合四种预训练模型同步完成材质分类与表面重建。
- Swin Transformer在分类上表现最佳(准确率89.21%),MAE在重建上最优(PSNR 16.06dB)。
- 所有模型均支持实时推理(<5ms/图),适合机器人感知等低算力场景。
从不完整的视觉观测中进行表面材质识别仍是机器人感知与环境理解中的挑战性问题。本文提出稀疏表面理解框架(SSUF),一种统一的双任务学习框架,将四种预训练架构——卷积自编码器(ConvAE)、视觉变换器(ViT)、Swin Transformer 和掩码自编码器(MAE)——适配于同时进行表面重建与材质分类。在 Touch-and-Go 数据集上,采用稀疏观测协议,仅保留原始图像的10%可见区域,其余部分被遮蔽。为保证公平比较,重建导向模型扩展了分类头,分类导向模型增加了重建解码器。评估指标包括重建质量、分类性能、模型复杂度与推理效率。实验结果表明各模型表现各异:Swin Transformer 在分类上表现最佳,准确率为89.21%,F1-score为0.8922,ROC-AUC为0.9813;而MAE在重建方面得分最高,PSNR达16.06 dB,SSIM为0.4501;ViT在重建与分类之间提供最佳平衡。此外,所有模型均可实现实时推理,单张图像处理时间低于5毫秒。总体表明,预训练模型可在极端视觉稀疏条件下支持材质识别,但高精度图像重建仍具挑战。
原文摘要 · Abstract (English)
Surface material recognition from incomplete visual observations remains a challenging problem in robotic perception and environmental understanding. This paper discusses Sparse Surface Understanding Framework (SSUF), a unified dual-task learning framework that adapts four pretrained architectures-Convolutional Autoencoder (ConvAE), Vision Transformer (ViT), Swin Transformer, and Masked Autoencoder (MAE) for si-multaneous surface reconstruction and material classification. Experiments were conducted on the Touch-and-Go dataset using a sparse observation protocol in which only 10% of the original image remained visible while the remaining regions were masked. To enable a fair comparison, reconstruction-oriented models were extended with classification heads, whereas classification- oriented models were augmented with reconstruction decoders. The resulting architectures were assessed using reconstruction quality, classification performance, model complexity, and in-ference efficiency metrics. Experimental results revealed distinct strengths across the models. Swin Transformer achieved the best classification performance with an accuracy of 89.21%, an F1-score of 0.8922, and a ROC-AUC of 0.9813. In contrast, MAE produced the highest reconstruction scores among evaluated models, with a PSNR of 16.06 dB and an SSIM of 0.4501, while ViT provided the best overall balance between reconstruction and classification performance. Furthermore, all models achieved real-time inference, requiring less than 5 ms per image. Over-all, the results show that pretrained architectures can support material recognition under severe visual sparsity, while accurate image reconstruction remains challenging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。