用强化学习让舞蹈生成更真实,避免穿模和脚滑
Skeleton2Stage: Reward-Guided Fine-Tuning for Physically Plausible Dance Generation
- 通过物理仿真设计奖励函数,引导扩散模型生成符合人体力学的舞蹈动作
- 在多个数据集上显著降低穿模和脚地偏差,生成动作更自然
- 适合关注真实感舞蹈生成的研究者与动画制作人员
尽管舞蹈生成技术已有进展,多数方法仍在骨骼空间训练,忽略网格层面的物理约束。导致看似合理的关节轨迹在人体网格可视化时出现身体自穿模和足地接触异常,影响观赏性并限制实际应用。本文通过从人体网格提取基于物理的奖励,采用强化学习微调(RLFT)引导扩散模型生成符合网格视觉的物理合理动作。奖励设计包含:(i) 仿效奖励,衡量动作在物理模拟器中的可模仿性(惩罚穿模与脚滑);(ii) 足地偏差(FGD)奖励,结合测试时的FGD引导以更好捕捉舞蹈中的动态足地交互。然而发现物理奖励常促使模型生成冻结动作以减少物理异常。为此提出抗冻结奖励,在保持物理合理性的同时保留运动动态性。在多个舞蹈数据集上的实验表明,本方法能显著提升生成动作的物理真实性,使舞蹈更逼真、更具美感。
原文摘要 · Abstract (English)
Despite advances in dance generation, most methods are trained in the skeletal domain and ignore mesh-level physical constraints. As a result, motions that look plausible as joint trajectories often exhibit body self-penetration and Foot-Ground Contact (FGC) anomalies when visualized with a human body mesh, reducing the aesthetic appeal of generated dances and limiting their real-world applications. We address this skeleton-to-mesh gap by deriving physics-based rewards from the body mesh and applying Reinforcement Learning Fine-Tuning (RLFT) to steer the diffusion model toward physically plausible motion synthesis under mesh visualization. Our reward design combines (i) an imitation reward that measures a motion's general plausibility by its imitability in a physical simulator (penalizing penetration and foot skating), and (ii) a Foot-Ground Deviation (FGD) reward with test-time FGD guidance to better capture the dynamic foot-ground interaction in dance. However, we find that the physics-based rewards tend to push the model to generate freezing motions for fewer physical anomalies and better imitability. To mitigate it, we propose an anti-freezing reward to preserve motion dynamics while maintaining physical plausibility. Experiments on multiple dance datasets consistently demonstrate that our method can significantly improve the physical plausibility of generated motions, yielding more realistic and aesthetically pleasing dances. The project page is available at: https://jjd1123.github.io/Skeleton2Stage/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。