聚焦关键步骤,提升掩码图像生成的强化学习效率
MaskFocus: Focusing Policy Optimization on Critical Steps for Masked Image Generation
- 通过中间图像与最终图像的相似度,识别生成过程中的关键步骤
- 仅对关键步骤优化策略,计算成本降低但生成质量不降
- 动态路由采样机制促进低熵样本探索更优掩码策略
强化学习在后训练语言模型和自回归视觉生成模型中展现巨大潜力,但将其应用于掩码生成模型仍具挑战。核心难点在于策略优化需考虑每一步的概率似然,因生成过程为多步迭代精炼,依赖完整采样轨迹带来高计算开销,而随机优化所有步骤常导致次优结果。本文提出MaskFocus,一种新型强化学习框架,通过聚焦关键步骤实现掩码生成模型的有效策略优化。具体地,通过测量每步中间图像与最终生成图像的相似度,确定步骤级信息增益,从而识别最关键、最有价值的步骤,并仅在这些步骤上执行策略优化。此外,设计基于熵的动态路由采样机制,鼓励低熵样本探索更优的掩码策略。在多个文本到图像基准上的大量实验验证了方法的有效性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has demonstrated significant potential for post-training language models and autoregressive visual generative models, but adapting RL to masked generative models remains challenging. The core factor is that policy optimization requires accounting for the probability likelihood of each step due to its multi-step and iterative refinement process. This reliance on entire sampling trajectories introduces high computational cost, whereas natively optimizing random steps often yields suboptimal results. In this paper, we present MaskFocus, a novel RL framework that achieves effective policy optimization for masked generative models by focusing on critical steps. Specifically, we determine the step-level information gain by measuring the similarity between the intermediate images at each sampling step and the final generated image. Crucially, we leverage this to identify the most critical and valuable steps and execute focused policy optimization on them. Furthermore, we design a dynamic routing sampling mechanism based on entropy to encourage the model to explore more valuable masking strategies for samples with low entropy. Extensive experiments on multiple Text-to-Image benchmarks validate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。