针对长文本视觉语言生成,提出分段奖励归一化方法提升生成质量。
SD-GRPO: Verifiable Segment Decomposition for Long-Form Vision-Language Generation

- 将输出按语义段分块,对每段独立计算归一化奖励
- 在多图密集描述任务中,段数越多效果越优,超越基线
- 适用于真实科学图表描述,尤其段间语义相关时需融合全局与分段奖励
Group Relative Policy Optimization(GRPO)及其变体虽在多模态大模型中表现优异,但其单标量优势分配机制难以适配长文本视觉语言任务。本文提出分段分解的GRPO(SD-GRPO),利用输出的自然分段结构,对每段进行z-score归一化的可验证奖励计算,生成向量形式的分段优势替代单一标量。在基于DOCCI构建的多面板密集描述任务中,段数越多,性能提升越显著;在基于MultiChartQA的多图表长问答任务中,理论与实证均表明整体奖励存在跨段误归因,且随输出长度增加而加剧。在真实世界科学图表描述任务(MMSci数据集)中,仅使用分段归一化仍不足,需结合全局与分段奖励。将SD-GRPO集成至Dr. GRPO框架后,验证其可零成本适配任意GRPO系统,显著提升长文本生成质量。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) and its variants, originally developed for Large Language Models (LLMs), have recently been applied to Multimodal LLMs and produced strong results. However, their coarse-grained holistic credit assignment from a single scalar advantage underfits vision-language (VL) tasks, where outputs are often long-form responses grounded in semantically rich images. To address this limitation, we exploit a structured signal that single-scalar formulations discard: the natural segmentation of long-form VL outputs. Concretely, we propose Segment-Decomposed GRPO (SD-GRPO), which z-normalizes verifiable per-segment rewards across the rollout group, yielding a vector of per-segment advantages in place of a single scalar. We evaluate SD-GRPO across three settings spanning controlled and real-world long-form VL generation, organized by increasing semantic entanglement across segments. On a controlled multi-panel dense-captioning task constructed from DOCCI, where segments are semantically independent, SD-GRPO consistently outperforms the GRPO baseline, with larger gains at higher segment counts. Extending to a controlled multi-chart long-form VQA task constructed from MultiChartQA, we show both theoretically and empirically that rollout-level rewards suffer from cross-segment credit misattribution that scales with output length. On a real-world scientific figure captioning task on the MMSci dataset, where subfigure captions share context across the figure, blending holistic and per-segment rewards further improves on both, suggesting per-segment normalization alone is insufficient when segments are semantically entangled. Finally, by integrating SD-GRPO into Dr. GRPO, we confirm that it can be applied to any GRPO framework with minimal implementation overhead to enhance long-form VL generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。