用信息增益奖励让医疗对话模型自学问诊，无需真人对话数据。
Aligning Medical Conversational AI through Online Reinforcement Learning with Information-Theoretic Rewards
- 基于信息增益的在线强化学习，让模型自动生成对话并优化提问策略。
- 在真实数据集上实现最高0.408的F1分数，比基础模型提升12.9%。
- 适合医疗AI研发者、临床辅助系统设计者，尤其关注多轮问诊场景。
我们提出信息增益微调（IGFT），一种新方法，用于训练医疗对话AI进行有效的患者问诊并生成全面的现病史（HPI），无需依赖预先收集的人类对话数据。IGFT结合在线组相对策略优化（GRPO）与信息论奖励，使模型能通过与模拟患者的自生成对话进行学习。不同于依赖昂贵专家标注对话或静态数据集的方法，我们的在线强化学习框架允许模型通过探索发现有效提问策略。核心创新在于信息增益奖励函数，可追踪对话中揭示的临床实体，如症状、时间模式和既往史。每个问题的奖励基于其预期信息增益，并结合GPT-4o-mini在临床相关性、患者参与度和具体性维度的评估。该混合方法确保模型学会提出有针对性且临床恰当的问题，高效获取诊断信息。我们使用LoRA微调两个模型：Llama-3.1-8B-Instruct 和 DeepSeek-R1-Distill-Qwen-7B（推理优化模型）。训练仅使用Avey数据集中的简洁现病史，评估其在包含更长、更详尽现病史的MIMIC数据集上的泛化能力。DeepSeek-R1-Distill-Qwen-7B（IGFT）在Avey上达到0.408的F1分数（较基线提升10.9%），在MIMIC上达0.289（提升12.9%）；而Llama-3.1-8B-Instruct（IGFT）分别达到0.384和0.336。两者均优于OpenAI模型，并超过针对单轮医疗问答优化的领域特定基线如HuatuoGPT和UltraMedical。
原文摘要 · Abstract (English)
We present Information Gain Fine-Tuning (IGFT), a novel approach for training medical conversational AI to conduct effective patient interviews and generate comprehensive History of Present Illness (HPI) without requiring pre-collected human conversations. IGFT combines online Group Relative Policy Optimization (GRPO) with information-theoretic rewards, enabling models to learn from self-generated conversations with simulated patients. Unlike existing approaches that rely on expensive expert-annotated conversations or static datasets, our online RL framework allows models to discover effective questioning strategies through exploration. Our key innovation is an information gain reward function that tracks which clinical entities such as symptoms, temporal patterns, and medical history, are revealed during conversation. Each question's reward is computed based on its expected information gain combined with GPT-4o-mini quality assessments across dimensions including clinical relevance, patient engagement, and specificity. This hybrid approach ensures models learn to ask targeted, clinically appropriate questions that efficiently gather diagnostic information. We fine-tune two models using LoRA: Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Qwen-7B (a reasoning-optimized model). Training exclusively on Avey data containing concise HPIs, we evaluate generalization to MIMIC data with longer, more elaborate HPIs. DeepSeek-R1-Distill-Qwen-7B (IGFT) achieves F1 scores of 0.408 on Avey (10.9% improvement over base) and 0.289 on MIMIC (12.9% improvement), while Llama-3.1-8B-Instruct (IGFT) reaches 0.384 and 0.336 respectively. Both models outperform OpenAI's model on MIMIC and surpass medical domain-specific baselines like HuatuoGPT and UltraMedical, which were optimized for single-turn medical QA rather than multi-turn conversations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。