传统机器学习在医学分类中仍优于大模型，微调效果差，零样本模型仅图像任务表现好。
LLM is Not All You Need: A Systematic Evaluation of ML vs. Foundation Models for text and image based Medical Classification
- 对比三种模型：经典机器学习、零样本大模型、轻量微调模型
- 经典模型在多数任务上表现最佳，尤其在结构化文本数据上
- 轻量微调（LoRA）效果最差，零样本模型仅图像任务有竞争力
本文通过四个公开的文本与图像数据集（涵盖二分类与多分类任务），系统评估了传统机器学习（ML）与基于Transformer的前沿模型在医学分类中的表现。对比了三类模型：经典机器学习（LR、LightGBM、ResNet-50）、提示驱动的大语言/视觉语言模型（Gemini 2.5）以及参数高效微调模型（LoRA适配的Gemma3）。所有实验采用一致数据划分与评估指标。结果显示，传统机器学习模型在大多数医疗分类任务中持续表现最优，尤其在结构化文本数据上优势显著。相反，LoRA微调的Gemma3变体在所有文本与图像任务中均表现最差，未能从极简微调中获益。而零样本的Gemini 2.5在文本任务上表现不佳，但在多分类图像任务上达到与经典ResNet-50相当的水平。结果表明，基础模型并非普遍更优，参数高效微调的效果高度依赖策略，本研究中极简微调反而导致性能下降。
原文摘要 · Abstract (English)
The combination of multimodal Vision-Language Models (VLMs) and Large Language Models (LLMs) opens up new possibilities for medical classification. This work offers a rigorous, unified benchmark by using four publicly available datasets covering text and image modalities (binary and multiclass complexity) that contrasts traditional Machine Learning (ML) with contemporary transformer-based techniques. We evaluated three model classes for each task: Classical ML (LR, LightGBM, ResNet-50), Prompt-Based LLMs/VLMs (Gemini 2.5), and Fine-Tuned PEFT Models (LoRA-adapted Gemma3 variants). All experiments used consistent data splits and aligned metrics. According to our results, traditional machine learning (ML) models set a high standard by consistently achieving the best overall performance across most medical categorization tasks. This was especially true for structured text-based datasets, where the classical models performed exceptionally well. In stark contrast, the LoRA-tuned Gemma variants consistently showed the worst performance across all text and image experiments, failing to generalize from the minimal fine-tuning provided. However, the zero-shot LLM/VLM pipelines (Gemini 2.5) had mixed results; they performed poorly on text-based tasks, but demonstrated competitive performance on the multiclass image task, matching the classical ResNet-50 baseline. These results demonstrate that in many medical categorization scenarios, established machine learning models continue to be the most reliable option. The experiment suggests that foundation models are not universally superior and that the effectiveness of Parameter-Efficient Fine-Tuning (PEFT) is highly dependent on the adaptation strategy, as minimal fine-tuning proved detrimental in this study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。