针对医学图像定位难问题,动态调整奖励标准提升训练效果。
MedLoc-R1: Performance-Aware Curriculum Reward Scheduling for GRPO-Based Medical Visual Grounding
- 根据模型能力自动调整奖励阈值,逐步收紧定位要求。
- 在三个医学数据集上定位精度显著优于传统GRPO方法。
- 无需额外网络,适合高风险医疗场景的可解释性分析。
医学视觉定位是细粒度多模态推理与可解释临床决策支持的关键基础。尽管强化学习在定位任务中取得进展,但现有基于组相对策略优化(GRPO)的方法在医学图像上面临严重奖励稀疏问题,主要源于小目标或模糊区域定位难度高,且固定交并比(IoU)奖励机制僵化低效,导致早期训练梯度消失、优化停滞。为此,我们提出MedLoc-R1,一种性能感知的奖励调度框架,根据模型进展逐步收紧奖励标准。该方法引入滑动窗口性能追踪器与多条件更新规则,实现从密集易得信号到严格精细定位要求的平滑过渡,同时保留GRPO优势,无需额外网络或梯度路径。在三个医学视觉定位基准上的实验表明,MedLoc-R1持续提升定位精度与训练稳定性。本框架为高风险医疗场景下的强化学习定位提供通用、轻量且高效解决方案。代码与模型检查点见https://github.com/MembrAI/MedLoc-R1。
原文摘要 · Abstract (English)
Medical visual grounding serves as a crucial foundation for fine-grained multimodal reasoning and interpretable clinical decision support. Despite recent advances in reinforcement learning (RL) for grounding tasks, existing approaches such as Group Relative Policy Optimization~(GRPO) suffer from severe reward sparsity when directly applied to medical images, primarily due to the inherent difficulty of localizing small or ambiguous regions of interest, which is further exacerbated by the rigid and suboptimal nature of fixed IoU-based reward schemes in RL. This leads to vanishing policy gradients and stagnated optimization, particularly during early training. To address this challenge, we propose MedLoc-R1, a performance-aware reward scheduling framework that progressively tightens the reward criterion in accordance with model readiness. MedLoc-R1 introduces a sliding-window performance tracker and a multi-condition update rule that automatically adjust the reward schedule from dense, easily obtainable signals to stricter, fine-grained localization requirements, while preserving the favorable properties of GRPO without introducing auxiliary networks or additional gradient paths. Experiments on three medical visual grounding benchmarks demonstrate that MedLoc-R1 consistently improves both localization accuracy and training stability over GRPO-based baselines. Our framework offers a general, lightweight, and effective solution for RL-based grounding in high-stakes medical applications. Code \& checkpoints are available at \hyperlink{}{https://github.com/MembrAI/MedLoc-R1}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。