用离散扩散模型提升图文交替推理效率，减少计算量同时保持性能。
Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Model
- 采用离散扩散模型实现局部视觉编辑，避免全图重生成。
- 相比自回归基线，推理计算减少26.9%，性能几乎无损。
- 提出解耦奖励分配，显著提升跨模态推理效果，适合多模态强化学习研究者。
基于强化学习的后训练已被广泛用于统一多模态模型中，以实现文本与图像生成任务中的图文交替推理。然而，现有方法大多基于自回归（AR）统一模型，导致视觉推理需完整重生成图像。本文表明，多模态离散扩散模型可作为替代方案，在交错推理中进行高效的视觉回溯，通过局部视觉编辑而非全图令牌重生成实现。相较于自回归基线，该方法在GRPO中将回溯计算减少26.9%，且性能损失极小。尽管效率提升，我们发现联合奖励分配（共享跨模态奖励信号）在强化学习更新中引入无关图像与文本序列间的交叉干扰。为此，我们提出解耦奖励分配策略，独立为文本与视觉片段分配奖励。采用该策略后，强化学习方法相较联合奖励分配提升11.2%，相较基础模型提升38.04%。
原文摘要 · Abstract (English)
RL-based post-training has been widely adopted to enable interleaved visual and textual reasoning in unified multimodal models capable of both text and image generation. However, most existing approaches are built upon autoregressive (AR) unified models, which require full image regeneration during visual reasoning. In this work, we demonstrate that multimodal discrete diffusion models are effective alternatives to AR models for reinforcement learning in interleaved reasoning, owing to their ability to perform efficient visual rollouts via localized visual editing rather than full image-token regeneration. This reduces rollout computation during GRPO by 26.9\% compared to AR baselines, with minimal performance drop. Despite the improved efficiency, we find that joint reward assignment, which employs a shared reward signal across modalities, introduces cross-modal interference between unrelated image and text token sequences during RL updates. To address this issue, we propose factorized reward assignment, a strategy that assigns rewards independently to text and vision segments. With factorized reward assignment, our RL approach achieves an 11.2% improvement over joint reward assignment and a 38.04% improvement over the base model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。