arXiv:2606.05174cs.CLcs.AI2026-06被引 1

用更智能的评分机制提升小模型的心脏病问答能力。

Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO

论文配图:Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO
图 1 · 摘自论文原文
  • 设计新奖励函数,利用多维度评分生成更精准优化信号
  • 在心脏病数据集上准确率从36.2%提至50.2%,F1升至66.8%
  • 适合想用小模型做医疗问答的研究者和开发者

大型语言模型在医疗应用中展现巨大潜力,但因数据隐私、推理成本及难以部署于边缘设备,实际落地困难。这促使开发更小高效的模型,并需强健的后训练策略确保可靠医学推理。本文研究基于RaR-Medicine的评分体系,采用组相对策略优化(GRPO)对心科医学问答任务进行后训练。提出方差感知奖励框架,将评分的显式与隐式聚合策略升级为基于各评价维度结果的连续分析奖励函数,取代传统的加权二元判断与单一李克特量表评分。该方法在稀疏、多维度且难自动验证的反馈中提供更丰富优化信号,实现更稳定的在线策略强化学习。在HealthBench的心脏相关子集上,最佳GRPO变体使准确率从0.362提升至0.502,F1从0.532升至0.668,接近GPT-OSS-120B(0.508准确率,0.674 F1)。结果表明,精心设计的基于评分标准的奖励机制是提升心科医学问答性能的有效路径,具备向其他评分驱动任务扩展的潜力。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have shown strong promise in healthcare applications. Yet deploying general-purpose models in real-world settings remains difficult due to data privacy constraints, inference costs, and limited suitability for edge or on-device use. These challenges motivate the development of smaller, more efficient models that require robust post-training strategies to ensure reliable medical reasoning. In this work, we investigate Group Relative Policy Optimization (GRPO) for post-training LLMs on heart-focused medical question answering with rubric-based supervision derived from RaR-Medicine. We propose a Variance-Aware Reward Framework that extends the Explicit Aggregation and Implicit Aggregation strategies of Rubrics as Rewards by replacing weighted binary criterion aggregation and single overall Likert-style scoring with continuous analytical reward functions derived from criterion-level rubric outcomes. This formulation provides richer optimization signals for feedback that is sparse, multi-criteria, and difficult to verify automatically, and enables more stable on-policy reinforcement learning. On a held-out heart-related subset of HealthBench, our best GRPO variant improves accuracy from 0.362 to 0.502 and F1 from 0.532 to 0.668 relative to the Qwen3-14B base model, while remaining competitive with GPT-OSS-120B (0.508 accuracy, 0.674 F1). Our findings show that carefully designed rubric-based rewards provide a practical strategy for improving heart-focused medical question answering in LLMs, with potential to extend to other rubric-based tasks.

医疗问答强化学习小模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。