arXiv:2606.26981cs.ROcs.AI2026-06

用语言模型规划动作,再通过物理模拟实时调整,让动作既符合指令又逼真。

In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics

论文配图:In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
图 1 · 摘自论文原文
  • 用大模型解析文本指令,生成候选动作序列
  • 结合物理仿真和语义对齐,选最优动作并迭代优化
  • 无需重新训练就能应对新指令,适合复杂场景

从文本描述生成人体动作对沉浸式数字应用至关重要,但现有方法在语义准确性和物理真实性之间存在权衡。基于大语言模型(LLM)的方法能理解开放词汇指令并生成高层动作规划,但常违反物理约束;物理感知模型虽提升真实性,却难以处理语义复杂性、细粒度指令和新概念。为此,我们提出上下文模型预测生成(ICMPG),将语言模型规划与推理时的物理反馈结合。ICMPG将动作生成重构为类似模型预测控制(MPC)的过程,包含两个模块:上下文感知动作生成(CAMG)模块利用LLM作为规划器,分解文本命令并从动作标记生成候选动作序列;模型预测生成(MPG)模块通过物理仿真和语义对齐评估候选序列,计算综合奖励,并选择最佳序列引导后续生成。与开环生成不同,该闭环优化使ICMPG能在不进行任务特定策略重训练的情况下,同时适应输入语义和模拟物理环境。在标准和零样本开放词汇设置下的大量实验表明,ICMPG能稳健泛化至多样化指令,在评估基准上生成的动作比代表性基线更符合物理规律且语义更忠实。该框架在语义理解与物理仿真间建立桥梁,同时保持灵活性,支持不同LLM主干,实现更通用、可控的文本驱动动作合成。

原文摘要 · Abstract (English)

Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and physical realism. Large language model (LLM)-based approaches can interpret diverse open-vocabulary instructions and compose high-level action plans, but they often generate motions that violate physical constraints. Physics-aware models improve realism through simulation or control, but they struggle with semantic complexity, fine-grained instructions, and novel concepts. To address this gap, we propose In-Context Model Predictive Generation (ICMPG), a framework that integrates language-model planning with inference-time physical feedback. ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules. The Context-Aware Motion Generation (CAMG) module uses an LLM as a planner to decompose textual commands and generate candidate motion sequences from motion tokens. The Model Predictive Generation (MPG) module evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence to guide subsequent generation steps. Unlike open-loop generation, this closed-loop refinement enables ICMPG to adapt motions to both the input semantics and the simulated physical environment without task-specific policy retraining. Extensive experiments across standard and zero-shot open-vocabulary settings show that ICMPG generalizes robustly to diverse commands and produces motions that are more physically plausible and semantically faithful than representative baselines on the evaluated benchmarks. The framework bridges semantic interpretation and physical simulation while remaining flexible enough to incorporate different LLM backbones, enabling more versatile and controllable text-driven motion synthesis.

动作生成语言模型物理模拟多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。