构建医学超声多模态数据集,提升大模型在专业领域的推理能力。
HAIBU-ReMUD: Reasoning Multimodal Ultrasound Dataset and Model Bridging to General Specific Domains
- 从分散的医学超声资料中自动生成图文问答四元组数据。
- 建成包含4.5万条数据的ReMUD数据集,显著提升模型在超声领域的表现。
- 适合医疗AI研究者和多模态大模型开发者使用。
多模态大语言模型在通用领域表现优异,但在特定领域(如医学超声)因缺乏领域专属数据而性能不佳。尽管医学超声领域存在大量诊断书籍、指南和报告等图文资料,但这些材料多以PDF、图片等形式分散存储,无法直接用于多模态大模型训练。本文提出一种新型图像-文本推理监督微调数据生成流程,从领域专有材料中构建包含图像、问题、思维链与答案的四元组数据。基于此构建了医学超声领域数据集ReMUD,包含超过45,000条推理与非推理型监督微调问答(QA)及视觉问答(VQA)数据。基于Qwen2.5-VL-7B-Instruct微调的ReMUD-7B模型,在医学超声任务上优于通用领域多模态大模型。为促进研究,本文将公开ReMUD数据集、数据生成代码及ReMUD-7B参数,网址为https://github.com/ShiDaizi/ReMUD,以缓解特定领域多模态大模型的数据短缺问题。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have shown great potential in general domains but perform poorly in some specific domains due to a lack of domain-specific data, such as image-text data or vedio-text data. In some specific domains, there is abundant graphic and textual data scattered around, but lacks standardized arrangement. In the field of medical ultrasound, there are ultrasonic diagnostic books, ultrasonic clinical guidelines, ultrasonic diagnostic reports, and so on. However, these ultrasonic materials are often saved in the forms of PDF, images, etc., and cannot be directly used for the training of MLLMs. This paper proposes a novel image-text reasoning supervised fine-tuning data generation pipeline to create specific domain quadruplets (image, question, thinking trace, and answer) from domain-specific materials. A medical ultrasound domain dataset ReMUD is established, containing over 45,000 reasoning and non-reasoning supervised fine-tuning Question Answering (QA) and Visual Question Answering (VQA) data. The ReMUD-7B model, fine-tuned on Qwen2.5-VL-7B-Instruct, outperforms general-domain MLLMs in medical ultrasound field. To facilitate research, the ReMUD dataset, data generation codebase, and ReMUD-7B parameters will be released at https://github.com/ShiDaizi/ReMUD, addressing the data shortage issue in specific domain MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。