对比LLM与CNN在脑肿瘤影像分类分割中的表现，发现前者空间理解差、效果不如后者。
A Comparison and Evaluation of Fine-tuned Convolutional Neural Networks to Large Language Models for Image Classification and Segmentation of Brain Tumors on MRI
- 用LLaMA 3.2和3D CNN在BraTS 2020数据集上比较脑肿瘤分类与分割性能
- CNN分类准确率80%，而原版LLM仅76%且特异度18%，微调后虽提升至55%但整体下降
- LLM输出集中于图像中心，无法区分肿瘤大小位置，适合研究者关注其局限性
大型语言模型（LLMs）在文本类医疗任务中表现优异，但在图像应用中的价值尚不明确。本文研究了通用视觉-语言模型LLaMA 3.2 Instruct在胶质瘤分类与分割任务中的表现，并与定制3D卷积神经网络（CNN）进行对比。基于BraTS 2020多模态脑部MRI数据集，实验发现：在低级别与高级别胶质瘤分类任务中，CNN达到80%准确率，且精确率与召回率平衡；而未微调的LLM准确率为76%，但特异度仅为18%，常将低级别肿瘤误判为高级别。微调后特异度提升至55%，但准确率降至72%。在分割任务中，采用中心点、边界框与多边形提取三种方法，CNN能准确定位肿瘤，小病灶偶有遗漏；而LLM预测始终聚集在图像中心，无法反映肿瘤实际大小、位置或分布。微调改善了输出格式，但未显著提升空间准确性，边界框法产生随机无结构输出。总体而言，CNN在两项任务中均优于LLM。当前形式的LLM在医学图像任务中存在严重空间理解缺陷，微调未能有效解决此问题，表明其尚不适合直接用于图像分析，未来需更严格的微调策略或新训练范式以提升其在医疗领域的适用性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown strong performance in text-based healthcare tasks. However, their utility in image-based applications remains unexplored. We investigate the effectiveness of LLMs for medical imaging tasks, specifically glioma classification and segmentation, and compare their performance to that of traditional convolutional neural networks (CNNs). Using the BraTS 2020 dataset of multi-modal brain MRIs, we evaluated a general-purpose vision-language LLM (LLaMA 3.2 Instruct) both before and after fine-tuning, and benchmarked its performance against custom 3D CNNs. For glioma classification (Low-Grade vs. High-Grade), the CNN achieved 80% accuracy and balanced precision and recall. The general LLM reached 76% accuracy but suffered from a specificity of only 18%, often misclassifying Low-Grade tumors. Fine-tuning improved specificity to 55%, but overall performance declined (e.g., accuracy dropped to 72%). For segmentation, three methods - center point, bounding box, and polygon extraction, were implemented. CNNs accurately localized gliomas, though small tumors were sometimes missed. In contrast, LLMs consistently clustered predictions near the image center, with no distinction of glioma size, location, or placement. Fine-tuning improved output formatting but failed to meaningfully enhance spatial accuracy. The bounding polygon method yielded random, unstructured outputs. Overall, CNNs outperformed LLMs in both tasks. LLMs showed limited spatial understanding and minimal improvement from fine-tuning, indicating that, in their current form, they are not well-suited for image-based tasks. More rigorous fine-tuning or alternative training strategies may be needed for LLMs to achieve better performance, robustness, and utility in the medical space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。