通过分步奖励机制，让医疗多模态模型纠正早期推理错误，提升诊断准确率。
Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning
- 引入分步奖励机制，对早期错误推理施加指数级惩罚。
- 将早期错误率从64.0%降至13.0%，答案准确率提升2.79点。
- 适用于需要可解释推理的临床AI系统，尤其适合小模型优化。
近期多模态大语言模型在临床图像推理中展现巨大潜力，但现有后训练流程仍以最终答案正确性或序列偏好为导向，导致信用分配稀疏，难以优化关键的推理过程。分析表明，早期推理失败引发的错误级联是医学视觉问答基准中预测错误的主要原因。为此，我们提出医疗推理感知策略优化（MRPO），一种融入分步过程奖励的强化学习算法。当最终答案错误时，MRPO对早期无效推理步骤中的标记施加指数级更大惩罚，有效打破错误级联，同时不损害成功路径。在三个多模态LLM骨干模型上，MRPO持续优于标准GRPO和近期强化学习基线；在Qwen3-VL-8B-Instruct上甚至超越更大的医疗多模态模型HuatuoGPT-Vision-34B达2.79分。此外，MRPO将早期推理错误率从64.0%降至13.0%，证明针对性缓解级联错误能显著提升推理质量与最终答案准确性。代码已开源。
原文摘要 · Abstract (English)
Recent multimodal large language models have shown great promise in clinical image reasoning, but existing post-training pipelines remain predominantly outcome-centric, relying on final answer correctness or sequence-level preferences. This suffers from sparse credit assignment, making it difficult to optimize the reasoning process essential for clinical applications. Our analysis reveals that cascading errors from early-stage reasoning failures are a leading cause of incorrect predictions in medical visual question answering (VQA) benchmarks. Motivated by this, we propose Medical Reasoning-aware Policy Optimization (MRPO), an RL algorithm that incorporates step-wise process rewards. When the final answer is incorrect, MRPO assigns exponentially larger penalties to tokens in earlier invalid reasoning steps, breaking failure cascades without compromising successful paths. Across three multimodal LLM backbones, MRPO consistently outperforms standard GRPO and a recent RL baseline, and on Qwen3-VL-8B-Instruct even surpasses substantially larger medical MLLMs such as HuatuoGPT-Vision-34B by 2.79 points. Moreover, MRPO reduces early-stage reasoning failures from 64.0% to 13.0%, showing that targeted mitigation of cascading failures improves both reasoning quality and final answer accuracy. Our code is available at https://github.com/dmis-lab/MRPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。