将医学影像中的手工特征与视觉变换器结合，提升分类准确率。
Embedding Radiomics into Vision Transformers for Multimodal Medical Image Classification
- 在视觉变换器中早期融合手工提取的影像组学特征与像素嵌入
- 在三个公开数据集上均达到当前最优表现，最高AUC达0.989
- 适合需要可解释性与高精度的医疗图像分类场景
背景：深度学习显著推动了医学图像分析，视觉变换器（ViTs）通过自注意力机制建模长距离依赖，成为卷积模型的有力替代。然而，ViTs对数据量要求高且缺乏领域特定先验，限制其在医学影像中的应用。相比之下，影像组学提供可解释的组织异质性描述符，但难以规模化且难融入端到端学习框架。本文提出一种融合影像组学的视觉变换器（RE-ViT），将影像组学特征与数据驱动的视觉嵌入在ViT主干中进行早期融合。目的：构建一个混合式RE-ViT框架，通过早期融合整合影像组学与分块视觉嵌入，增强医学图像分类的鲁棒性与性能。方法：遵循标准ViT流程，图像被划分为图像块。对每个图像块，提取手工影像组学特征并与线性投影后的像素嵌入融合。融合表示经归一化、位置编码后输入ViT编码器。可学习[CLS]标记聚合各块信息用于分类。在三个公开数据集（包括BUSI、ChestXray2017和Retinal OCT）上评估，指标为准确率、宏AUC、敏感性和特异性。与基于CNN（VGG-16、ResNet）及混合模型（TransMed）对比。结果：RE-ViT取得领先性能：在BUSI上AUC=0.950±0.011；在ChestXray2017上AUC=0.989±0.004；在Retinal OCT上AUC=0.986±0.001，优于所有对比模型。结论：RE-ViT框架有效整合影像组学与ViT架构，在多模态医学图像分类任务中表现出更优性能与泛化能力。
原文摘要 · Abstract (English)
Background: Deep learning has significantly advanced medical image analysis, with Vision Transformers (ViTs) offering a powerful alternative to convolutional models by modeling long-range dependencies through self-attention. However, ViTs are inherently data-intensive and lack domain-specific inductive biases, limiting their applicability in medical imaging. In contrast, radiomics provides interpretable, handcrafted descriptors of tissue heterogeneity but suffers from limited scalability and integration into end-to-end learning frameworks. In this work, we propose the Radiomics-Embedded Vision Transformer (RE-ViT) that combines radiomic features with data-driven visual embeddings within a ViT backbone. Purpose: To develop a hybrid RE-ViT framework that integrates radiomics and patch-wise ViT embeddings through early fusion, enhancing robustness and performance in medical image classification. Methods: Following the standard ViT pipeline, images were divided into patches. For each patch, handcrafted radiomic features were extracted and fused with linearly projected pixel embeddings. The fused representations were normalized, positionally encoded, and passed to the ViT encoder. A learnable [CLS] token aggregated patch-level information for classification. We evaluated RE-ViT on three public datasets (including BUSI, ChestXray2017, and Retinal OCT) using accuracy, macro AUC, sensitivity, and specificity. RE-ViT was benchmarked against CNN-based (VGG-16, ResNet) and hybrid (TransMed) models. Results: RE-ViT achieved state-of-the-art results: on BUSI, AUC=0.950+/-0.011; on ChestXray2017, AUC=0.989+/-0.004; on Retinal OCT, AUC=0.986+/-0.001, which outperforms other comparison models. Conclusions: The RE-ViT framework effectively integrates radiomics with ViT architectures, demonstrating improved performance and generalizability across multimodal medical image classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。