用大模型推理+物理优化,让文字生成动作更自然真实
Re$^2$MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement

- 用大模型加蒙特卡洛搜索生成关键帧动作规划
- 通过动态时间匹配和物理奖励提升动作完整度与真实性
- 适合需要灵活生成新动作的动画、游戏开发者
文本到动作(T2M)生成旨在通过文本描述控制角色行为。现有方法在训练数据分布内表现优异,但在面对与训练文本差异较大的描述时性能显著下降。为此,我们提出Re$^2$MoGen框架,利用增强型大语言模型(LLM)推理生成初始动作规划,并通过强化学习后训练进行物理合理性优化。该框架包含三阶段:首先采用蒙特卡洛树搜索提升LLM在文本提示下生成合理关键帧的能力,仅指定根节点及部分关键关节位置以简化推理;其次,使用人体姿态模型作为先验,基于规划的关键帧优化全身姿态,并以生成的不完整动作监督预训练动作生成器的微调,通过动态时间匹配目标实现时空补全;最后,通过物理感知奖励进行后训练优化,消除LLM生成动作中的物理不合理性。大量实验表明,该框架可生成语义一致且物理合理的动作,在开放词汇动作生成任务中达到领先水平。
原文摘要 · Abstract (English)
Text-to-motion (T2M) generation aims to control the behavior of a target character via textual descriptions. Leveraging text-motion paired datasets, existing T2M models have achieved impressive performance in generating high-quality motions within the distribution of their training data. However, their performance deteriorates notably when the motion descriptions differ significantly from the training texts. To address this issue, we propose Re$^2$MoGen, a Reasoning and Refinement open-vocabulary Motion Generation framework that leverages enhanced Large Language Model (LLM) reasoning to generate an initial motion planning and then refine its physical plausibility via reinforcement learning (RL) post-training. Specifically, Re$^2$MoGen consists of three stages: We first employ Monte Carlo tree search to enhance the LLM's reasoning ability in generating reasonable keyframes of the motion based on text prompts, specifying only the root and several key joints' positions to ease the reasoning process. Then, we apply a human pose model as a prior to optimize the full-body poses based on the planned keyframes and use the resulting incomplete motion to supervise fine-tuning a pre-trained motion generator via a dynamic temporal matching objective, enabling spatiotemporal completion. Finally, we use post-training with physics-aware reward to refine motion quality to eliminate physical implausibility in LLM-planned motions. Extensive experiments demonstrate that our framework can generate semantically consistent and physically plausible motions and achieve state-of-the-art performance in open-vocabulary motion generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。