用知识增强生成医学推理数据,提升罕见病诊断能力
Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach

- 用罕见病知识合成可控分布的推理题
- 自动生成伪标签实现两阶段强化训练,最高提升5.93%
- 无需昂贵推理链蒸馏,适合医疗AI研发者
大语言模型在复杂医疗应用中潜力巨大,但高质量推理数据稀缺制约其发展。现有方法通常通过监督微调从大型专有模型中提炼思维链,再进行强化学习(RL),但在罕见病等低频领域改进有限,且生成复杂推理链成本高昂。为此,我们提出MedSSR:一种医学知识增强的数据合成与半监督强化学习框架。该框架首先利用罕见病知识合成可控制分布的推理问题;然后利用策略模型自身生成高质量伪标签,实现由内而外的两阶段训练:先在伪标注合成数据上进行自监督强化学习,再在人工标注真实数据上进行监督强化学习。该方法高效扩展模型训练,无需依赖高成本的推理链蒸馏。在Qwen和Llama上的大量实验表明,我们的方法在十个医学基准测试中均优于现有方法,在罕见病任务上最高提升5.93%。代码已公开于https://github.com/tdlhl/MedSSR。
原文摘要 · Abstract (English)
While large language models hold promise for complex medical applications, their development is hindered by the scarcity of high-quality reasoning data. To address this issue, existing approaches typically distill chain-of-thought reasoning traces from large proprietary models via supervised fine-tuning, then conduct reinforcement learning (RL). These methods exhibit limited improvement on underrepresented domains like rare diseases while incurring substantial costs from generating complex reasoning chains. To efficiently enhance medical reasoning, we propose MedSSR, a Medical Knowledge-enhanced data Synthesis and Semi-supervised Reinforcement learning framework. Our framework first employs rare disease knowledge to synthesize distribution-controllable reasoning questions. We then utilize the policy model itself to generate high-quality pseudo-labels. This enables a two-stage, intrinsic-to-extrinsic training paradigm: self-supervised RL on the pseudo-labeled synthetic data, followed by supervised RL on the human-annotated real data. MedSSR scales model training efficiently without relying on costly trace distillation. Extensive experiments on Qwen and Llama demonstrate that our method outperforms existing methods across ten medical benchmarks, achieving up to +5.93% gain on rare-disease tasks. Our code is available at https://github.com/tdlhl/MedSSR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。