通过细粒度奖励分配,让模型画图推理更精准。
SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More
- 用画标记的方式在图上逐步推理,自反馈优化每一步
- 在图表、图像和数学任务上平均提升7.23%
- 适合需要复杂视觉推理的智能分析系统
图表是承载复杂数据的高密度视觉信息载体,也是信息提取与分析的重要媒介。由于需要精确且复杂的视觉推理,现有多模态大语言模型(MLLMs)在自动化图表理解方面面临重大挑战。许多基于强化学习(RL)训练的MLLM存在信用分配难题:其优势估计通常在轨迹层面进行,无法区分单个生成响应中正确与错误的推理步骤。为解决这一问题,我们提出SketchVL,一种采用新型强化学习算法FinePO优化的MLLM,该算法支持轨迹内的细粒度信用分配。SketchVL的方法是在图像上绘制中间推理步骤作为标记,并将标注后的图像反馈给自身,形成稳健的多步推理流程。训练时,FinePO利用细粒度过程奖励模型(FinePRM)对轨迹内每个绘图动作打分,从而精确分配每一步的信用。该机制能在整体轨迹成功时强化正确标记,在整体表现不佳时重罚错误标记,实现细粒度强化信号。实验表明,SketchVL能使其步骤级行为与FinePRM对齐,在图表数据集、自然图像数据集及数学任务上,平均性能比基线模型提升7.23%,为构建强大推理模型提供了新方向。
原文摘要 · Abstract (English)
Charts are high-density visual carriers of complex data and medium for information extraction and analysis. Due to the need for precise and complex visual reasoning, automated chart understanding poses a significant challenge to existing Multimodal Large Language Models (MLLMs). Many MLLMs trained with reinforcement learning (RL) face the challenge of credit assignment. Their advantage estimation, typically performed at the trajectory level, cannot distinguish between correct and incorrect reasoning steps within a single generated response. To address this limitation, we introduce SketchVL, a novel MLLM that optimized with FinePO, a new RL algorithm designed for fine-grained credit assignment within each trajectory. SketchVL's methodology involves drawing its intermediate reasoning steps as markers on the image and feeding the annotated image back to itself, creating a robust, multi-step reasoning process. During training, the FinePO algorithm leverages a Fine-grained Process Reward Model (FinePRM) to score each drawing action within a trajectory, thereby precisely assigning credit for each step. This mechanism allows FinePO to more strongly reward correct tokens when a trajectory is globally successful, and more heavily penalize incorrect tokens when the trajectory is globally suboptimal, thus achieving fine-grained reinforcement signals. Experiments show that SketchVL learns to align its step-level behavior with the FinePRM, achieving an average performance gain of 7.23\% over its base model across chart datasets, natural image datasets, and mathematics, providing a promising new direction for training powerful reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。