小块尺寸提升医学图像分类性能，3D任务效果更显著。
Effect of Patch Size on Fine-Tuning Vision Transformers in Two-Dimensional and Three-Dimensional Medical Image Classification
- 用不同块大小微调ViT，发现1-4的小块尺寸表现最佳。
- 2D数据块大小从28减到2，平衡准确率提升12.78%。
- 融合1/2/4块模型预测，2D任务性能进一步提升。
视觉变换器（ViTs）及其变体已成为众多计算机视觉任务的前沿方法，并广泛用作大规模视觉与视觉-语言基础模型的骨干网络。尽管大量研究聚焦于架构改进，但作为ViT初始设计关键选择的块大小影响仍缺乏深入探索，尤其在同时存在二维（2D）和三维（3D）成像模态的医学领域。本研究基于12个来自不同成像模态的医学图像数据集（包括7个2D和5个3D数据集），系统评估了不同块大小对ViT分类性能的影响。使用单个GPU和多种块大小（1, 2, 4, 7, 14, 28）进行微调，结果显示较小块大小（1, 2, 4）在几乎所有数据集上均带来一致性能提升。具体而言，2D数据集的平衡准确率最高提升12.78%（块大小2对比28），3D数据集最高提升23.78%（块大小1对比14），代价是计算开销增加。此外，通过简单集成策略融合块大小为1、2、4的模型预测，多数情况下性能进一步提升，尤其在2D数据集上表现突出。代码已公开于GitHub：https://github.com/HealMaDe/MedViT
原文摘要 · Abstract (English)
Vision Transformers (ViTs) and their variants have become state-of-the-art in many computer vision tasks and are widely used as backbones in large-scale vision and vision-language foundation models. While substantial research has focused on architectural improvements, the impact of patch size, a crucial initial design choice in ViTs, remains underexplored, particularly in medical domains where both two-dimensional (2D) and three-dimensional (3D) imaging modalities exist. In this study, using 12 medical imaging datasets from various imaging modalities (including seven 2D and five 3D datasets), we conduct a thorough evaluation of how different patch sizes affect ViT classification performance. Using a single graphical processing unit (GPU) and a range of patch sizes (1, 2, 4, 7, 14, 28), we fine-tune ViT models and observe consistent improvements in classification performance with smaller patch sizes (1, 2, and 4), which achieve the best results across nearly all datasets. More specifically, our results indicate improvements in balanced accuracy of up to 12.78% for 2D datasets (patch size 2 vs. 28) and up to 23.78% for 3D datasets (patch size 1 vs. 14), at the cost of increased computational expense. Moreover, by applying a straightforward ensemble strategy that fuses the predictions of the models trained with patch sizes 1, 2, and 4, we demonstrate a further boost in performance in most cases, especially for the 2D datasets. Our implementation is publicly available on GitHub: https://github.com/HealMaDe/MedViT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。