用大模型+模块化网络实现医疗影像的指令式分析,无需重训新任务。
MedPrompt: LLM-CNN Fusion with Weight Routing for Medical Image Segmentation and Classification
- 大模型解析用户指令,动态路由专用权重完成任务
- 19个数据集上端到端正确率达97%,推理仅2.5秒
- 适合需要灵活部署的临床场景,兼顾准确与效率
当前医疗图像分析系统多为任务专用,需分别构建分类与分割模型,且难以支持用户自定义流程。为此,我们提出MedPrompt,一个统一框架:利用少量样本提示的大语言模型(Llama-4-17B)进行高层任务规划,结合模块化卷积神经网络(DeepFusionLab)执行低层图像处理。LLM解读用户指令并生成结构化输出,动态路由特定任务的预训练权重。该权重路由机制在新增任务时无需重训整个框架,仅需加载任务专属权重,显著提升可扩展性与部署效率。我们在19个公开数据集上评估,覆盖12项任务及5种成像模态。系统在解释与执行指令方面达到97%的端到端正确率,平均推理延迟为2.5秒,适用于近实时应用。DeepFusionLab在肺部分割中取得Dice系数0.9856,在结核病分类中达F1分数0.9744。整体上,MedPrompt通过融合大模型的可解释性与模块化CNN的高效性,实现了可扩展的指令驱动医疗影像分析。
原文摘要 · Abstract (English)
Current medical image analysis systems are typically task-specific, requiring separate models for classification and segmentation, and lack the flexibility to support user-defined workflows. To address these challenges, we introduce MedPrompt, a unified framework that combines a few-shot prompted Large Language Model (Llama-4-17B) for high-level task planning with a modular Convolutional Neural Network (DeepFusionLab) for low-level image processing. The LLM interprets user instructions and generates structured output to dynamically route task-specific pretrained weights. This weight routing approach avoids retraining the entire framework when adding new tasks-only task-specific weights are required, enhancing scalability and deployment. We evaluated MedPrompt across 19 public datasets, covering 12 tasks spanning 5 imaging modalities. The system achieves a 97% end-to-end correctness in interpreting and executing prompt-driven instructions, with an average inference latency of 2.5 seconds, making it suitable for near real-time applications. DeepFusionLab achieves competitive segmentation accuracy (e.g., Dice 0.9856 on lungs) and strong classification performance (F1 0.9744 on tuberculosis). Overall, MedPrompt enables scalable, prompt-driven medical imaging by combining the interpretability of LLMs with the efficiency of modular CNNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。