arXiv:2506.21199cs.CVeess.SP2025-06被引 2

用大模型+模块化网络实现医疗影像的指令式分析,无需重训新任务。

MedPrompt: LLM-CNN Fusion with Weight Routing for Medical Image Segmentation and Classification

  • 大模型解析用户指令,动态路由专用权重完成任务
  • 19个数据集上端到端正确率达97%,推理仅2.5秒
  • 适合需要灵活部署的临床场景,兼顾准确与效率

当前医疗图像分析系统多为任务专用,需分别构建分类与分割模型,且难以支持用户自定义流程。为此,我们提出MedPrompt,一个统一框架:利用少量样本提示的大语言模型(Llama-4-17B)进行高层任务规划,结合模块化卷积神经网络(DeepFusionLab)执行低层图像处理。LLM解读用户指令并生成结构化输出,动态路由特定任务的预训练权重。该权重路由机制在新增任务时无需重训整个框架,仅需加载任务专属权重,显著提升可扩展性与部署效率。我们在19个公开数据集上评估,覆盖12项任务及5种成像模态。系统在解释与执行指令方面达到97%的端到端正确率,平均推理延迟为2.5秒,适用于近实时应用。DeepFusionLab在肺部分割中取得Dice系数0.9856,在结核病分类中达F1分数0.9744。整体上,MedPrompt通过融合大模型的可解释性与模块化CNN的高效性,实现了可扩展的指令驱动医疗影像分析。

原文摘要 · Abstract (English)

Current medical image analysis systems are typically task-specific, requiring separate models for classification and segmentation, and lack the flexibility to support user-defined workflows. To address these challenges, we introduce MedPrompt, a unified framework that combines a few-shot prompted Large Language Model (Llama-4-17B) for high-level task planning with a modular Convolutional Neural Network (DeepFusionLab) for low-level image processing. The LLM interprets user instructions and generates structured output to dynamically route task-specific pretrained weights. This weight routing approach avoids retraining the entire framework when adding new tasks-only task-specific weights are required, enhancing scalability and deployment. We evaluated MedPrompt across 19 public datasets, covering 12 tasks spanning 5 imaging modalities. The system achieves a 97% end-to-end correctness in interpreting and executing prompt-driven instructions, with an average inference latency of 2.5 seconds, making it suitable for near real-time applications. DeepFusionLab achieves competitive segmentation accuracy (e.g., Dice 0.9856 on lungs) and strong classification performance (F1 0.9744 on tuberculosis). Overall, MedPrompt enables scalable, prompt-driven medical imaging by combining the interpretability of LLMs with the efficiency of modular CNNs.

医疗影像大模型指令驱动模块化网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。