用强化学习提升生成模型的可控性与真实感
Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
- 将强化学习引入生成模型，优化难以用传统损失函数衡量的目标
- 在图像、视频和3D生成中显著提升语义准确性和时序一致性
- 适合对生成内容质量有高要求的研究者与应用开发者
生成模型在合成图像、视频及3D/4D结构方面取得显著进展，但通常采用似然或重构损失等代理目标训练，常与感知质量、语义准确性或物理真实性存在偏差。强化学习（RL）为优化非可微、偏好驱动和时序结构化目标提供了原则性框架。近期研究表明，其在提升生成任务的可控性、一致性和人类对齐方面效果显著。本综述系统梳理了基于强化学习的视觉内容生成方法，回顾了从经典控制到通用优化工具的演进，分析其在图像、视频及3D/4D生成中的集成。在各领域中，强化学习不仅作为微调手段，更作为实现复杂高层目标对齐的结构性组件。最后，总结了该交叉领域的开放挑战与未来方向。
原文摘要 · Abstract (English)
Generative models have made significant progress in synthesizing visual content, including images, videos, and 3D/4D structures. However, they are typically trained with surrogate objectives such as likelihood or reconstruction loss, which often misalign with perceptual quality, semantic accuracy, or physical realism. Reinforcement learning (RL) offers a principled framework for optimizing non-differentiable, preference-driven, and temporally structured objectives. Recent advances demonstrate its effectiveness in enhancing controllability, consistency, and human alignment across generative tasks. This survey provides a systematic overview of RL-based methods for visual content generation. We review the evolution of RL from classical control to its role as a general-purpose optimization tool, and examine its integration into image, video, and 3D/4D generation. Across these domains, RL serves not only as a fine-tuning mechanism but also as a structural component for aligning generation with complex, high-level goals. We conclude with open challenges and future research directions at the intersection of RL and generative modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。