通过分阶段推理修正,让小模型诊断准确率提升12.7%。
MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction

- 用大模型筛选数据并注入医学知识,建立学生模型的知识基础。
- 两次微调后,小模型在胃肠科数据集上达56.4%准确率。
- 适合资源受限场景下需要高可靠诊断的小模型应用。
精准临床诊断依赖大量领域知识和复杂推理能力。尽管大语言模型(LLMs)在临床推理方面潜力巨大,但其高计算与内存开销限制了在资源受限环境中的部署。知识蒸馏(KD)可将大模型能力压缩至小模型,但传统方法仅传递表面答案模式,无法保留可靠诊断所需的结构化推理。为此,我们提出两阶段蒸馏框架MedThink,旨在培养小语言模型(SLMs)的稳健临床推理能力。第一阶段,教师模型筛选数据并注入领域知识解释,微调学生模型,建立知识基础。第二阶段,教师评估学生错误,生成连接知识与正确答案的推理链,并通过二次微调优化学生诊断推理。我们在通用医学基准和包含955个问答对的胃肠科数据集上评估MedThink。实验表明,它在所有基准上均优于六种蒸馏策略:通用任务中相较学生基线最高提升12.7%,胃肠科评估总准确率达56.4%。结果表明,以推理为核心的迭代蒸馏能显著提升小模型的诊断准确率与泛化能力,同时保持计算效率。代码与数据已公开于https://github.com/destinybird/PrecisionBoost。
原文摘要 · Abstract (English)
Accurate clinical diagnosis requires extensive domain knowledge and complex clinical reasoning capabilities. Although large language models (LLMs) hold great potential for clinical reasoning, their high computational and memory requirements limit their deployment in resource-constrained environments. Knowledge distillation (KD) can compress LLM capabilities into smaller models, but traditional KD merely transfers superficial answer patterns and fails to preserve the structured reasoning required for reliable diagnosis. To address this, we propose a two-stage distillation framework, MedThink, designed to cultivate robust clinical reasoning in small language models (SLMs). In the first stage, a teacher LLM screens data and injects domain-knowledge explanations to fine-tune a student model, establishing a knowledge foundation. In the second stage, the teacher evaluates the student's errors, generates reasoning chains linking knowledge to correct answers, and refines the student's diagnostic reasoning through a second round of fine-tuning. We evaluate MedThink on general medical benchmarks and a gastroenterology dataset comprising 955 question-answer pairs. Experiments demonstrate that MedThink outperforms six distillation strategies in all benchmarks: achieving an improvement of up to 12.7% over the student baseline in general tasks, and reaching a total top accuracy of 56.4% in gastroenterology evaluation. This indicates that iterative distillation centered on reasoning can significantly enhance the diagnostic accuracy and generalization capabilities of SLMs whilst maintaining computational efficiency. Our code and data are publicly available at https://github.com/destinybird/PrecisionBoost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。