用RGB图推断3D形状点云，提升植物标本图像分类准确率
SIM-Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds from RGB Images for 2D Classification
- 从2D图像掩码生成3D点云，融合纹理与几何特征
- 在标本数据集上准确率比ResNet101高9.9%，F-score提升12.3%
- 适合处理背景杂、遮挡多的生物图像分类任务
我们提出形状-图像多模态网络（SIM-Net），一种将从RGB图像直接推断出的3D点云表示融入2D图像分类的新架构。核心创新在于像素到点的转换机制，将2D物体掩码转化为3D点云，实现纹理与几何特征的融合，从而提升分类性能。该方法特别适用于数字化植物标本分类任务，可应对异质背景、非植物干扰物和遮挡等挑战。SIM-Net采用基于分割的预处理步骤提取物体掩码，再生成3D点云；其结构包含用于2D图像特征的CNN编码器和用于几何特征的PointNet编码器，二者在统一潜在空间中融合。在标本数据集上的实验表明，SIM-Net持续优于ResNet101，准确率最高提升9.9%，F-score提升12.3%；同时超越多个基于Transformer的前沿模型，验证了引入3D结构推理对2D分类任务的价值。
原文摘要 · Abstract (English)
We introduce the Shape-Image Multimodal Network (SIM-Net), a novel 2D image classification architecture that integrates 3D point cloud representations inferred directly from RGB images. Our key contribution lies in a pixel-to-point transformation that converts 2D object masks into 3D point clouds, enabling the fusion of texture-based and geometric features for enhanced classification performance. SIM-Net is particularly well-suited for the classification of digitized herbarium specimens (a task made challenging by heterogeneous backgrounds), non-plant elements, and occlusions that compromise conventional image-based models. To address these issues, SIM-Net employs a segmentation-based preprocessing step to extract object masks prior to 3D point cloud generation. The architecture comprises a CNN encoder for 2D image features and a PointNet-based encoder for geometric features, which are fused into a unified latent space. Experimental evaluations on herbarium datasets demonstrate that SIM-Net consistently outperforms ResNet101, achieving gains of up to 9.9% in accuracy and 12.3% in F-score. It also surpasses several transformer-based state-of-the-art architectures, highlighting the benefits of incorporating 3D structural reasoning into 2D image classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。