构建眼科专用多模态大模型，实现定位与诊断协同推理。
Constructing Ophthalmic MLLM for Positioning-diagnosis Collaboration Through Clinical Cognitive Chain Reasoning
- 通过临床认知链设计，整合局部定位与全局诊断推理。
- 在问答任务中准确率比40B MedRegA高26.6%，零样本报告生成临床一致性达77.0%。
- 自建数据集支持高质量标注，提升模型训练效率与可解释性。
多模态大语言模型（MLLM）在医学诊断中展现出巨大潜力，但在眼科等专业领域仍面临标注粒度碎片化与临床推理逻辑不一致的问题，阻碍跨模态理解的精准性。本文提出专用于眼科的FundusExpert模型，结合由智能眼底引擎（Fundus-Engine）构建的FundusGen数据集。该引擎自动完成病灶定位，并利用基于MLLM的语义扩展，将全局疾病分类、局部目标检测与细粒度特征分析统一于单张眼底图像中。通过构建符合临床逻辑的认知链，引导模型生成可解释的推理路径。Fine-tuned后的FundusExpert在眼科问答任务中表现最佳，平均准确率超越40B MedRegA 26.6%；在零样本报告生成任务中，临床一致性达77.0%，显著优于GPT-4o的47.6%。我们还揭示了数据质量与模型能力间的缩放规律（$L /propto N^{0.068}$），表明认知对齐标注可提升数据利用效率。本工作通过区域定位与诊断推理链的融合，构建了一个可扩展、临床对齐的MLLM，探索了特定领域视觉-语言鸿沟的解决路径。项目开源地址：https://github.com/MeteorElf/FundusExpert。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) demonstrate significant potential in the field of medical diagnosis. However, they face critical challenges in specialized domains such as ophthalmology, particularly the fragmentation of annotation granularity and inconsistencies in clinical reasoning logic, which hinder precise cross-modal understanding. This paper introduces FundusExpert, an ophthalmology-specific MLLM with integrated positioning-diagnosis reasoning capabilities, along with FundusGen, a dataset constructed through the intelligent Fundus-Engine system. Fundus-Engine automates localization and leverages MLLM-based semantic expansion to integrate global disease classification, local object detection, and fine-grained feature analysis within a single fundus image. Additionally, by constructing a clinically aligned cognitive chain, it guides the model to generate interpretable reasoning paths. FundusExpert, fine-tuned with instruction data from FundusGen, achieves the best performance in ophthalmic question-answering tasks, surpassing the average accuracy of the 40B MedRegA by 26.6%. It also excels in zero-shot report generation tasks, achieving a clinical consistency of 77.0%, significantly outperforming GPT-4o's 47.6%. Furthermore, we reveal a scaling law between data quality and model capability ($L \propto N^{0.068}$), demonstrating that the cognitive alignment annotations in FundusGen enhance data utilization efficiency. By integrating region-level localization with diagnostic reasoning chains, our work develops a scalable, clinically-aligned MLLM and explores a pathway toward bridging the visual-language gap in specific MLLMs. Our project can be found at https://github.com/MeteorElf/FundusExpert.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。