首次系统探索强化学习在文本生成3D模型中的应用,解决几何与纹理一致性难题。
Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation
- 设计多维度奖励机制,利用通用多模态模型提升3D属性生成信号。
- 提出分层优化算法Hi-GRPO,实现从粗略形状到细节纹理的渐进式改进。
- 构建新基准MME-3DR,评估3D生成模型的隐含推理能力,适合研究者参考。
强化学习(RL)在大语言模型和多模态模型中已证明有效,近期也被成功应用于2D图像生成。然而,由于3D物体具有更高的空间复杂性,需全局几何一致性和精细局部纹理,因此将RL应用于3D生成仍处于未充分探索阶段。本文首次系统研究了自回归式文本到3D生成中强化学习的应用,涵盖四个维度:(1) 奖励设计:评估不同奖励维度与模型选择,表明与人类偏好对齐至关重要,通用多模态模型能提供鲁棒的3D属性信号;(2) 强化学习算法:研究GRPO变体,突出令牌级优化的有效性,并进一步探究训练数据量与迭代次数的缩放效应;(3) 文本到3D基准:鉴于现有基准无法衡量3D生成模型的隐式推理能力,引入MME-3DR;(4) 高级强化学习范式:受3D生成自然层级结构启发,提出Hi-GRPO,通过专用奖励集合实现从全局到局部的分层优化。基于这些发现,我们开发出AR3D-R1,首个增强型强化学习驱动的文本到3D生成模型,可实现从粗略形状到纹理细节的逐步精炼。代码已开源于https://github.com/Ivan-Tang-3D/3DGen-R1。
原文摘要 · Abstract (English)
Reinforcement learning (RL), earlier proven to be effective in large language and multi-modal models, has been successfully extended to enhance 2D image generation recently. However, applying RL to 3D generation remains largely unexplored due to the higher spatial complexity of 3D objects, which require globally consistent geometry and fine-grained local textures. This makes 3D generation significantly sensitive to reward designs and RL algorithms. To address these challenges, we conduct the first systematic study of RL for text-to-3D autoregressive generation across several dimensions. (1) Reward designs: We evaluate reward dimensions and model choices, showing that alignment with human preference is crucial, and that general multi-modal models provide robust signal for 3D attributes. (2) RL algorithms: We study GRPO variants, highlighting the effectiveness of token-level optimization, and further investigate the scaling of training data and iterations. (3) Text-to-3D Benchmarks: Since existing benchmarks fail to measure implicit reasoning abilities in 3D generation models, we introduce MME-3DR. (4) Advanced RL paradigms: Motivated by the natural hierarchy of 3D generation, we propose Hi-GRPO, which optimizes the global-to-local hierarchical 3D generation through dedicated reward ensembles. Based on these insights, we develop AR3D-R1, the first RL-enhanced text-to-3D model, expert from coarse shape to texture refinement. We hope this study provides insights into RL-driven reasoning for 3D generation. Code is released at https://github.com/Ivan-Tang-3D/3DGen-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。