arXiv:2412.15664cs.CV2024-12被引 14

用自然语言控制人在复杂场景中行走,还能自适应地形变化。

SCENIC: Scene-aware Semantic Navigation with Instruction-guided Control

  • 分层场景理解:用目标导向的规范和局部距离场双机制建模环境
  • 支持任意长度动作生成,能响应如'小心跨过障碍'等文本指令
  • 可推广到四个真实场景数据集,适合虚拟人、游戏与动画领域

生成能适应复杂环境并支持自然语言控制的自然人类动作,仍是动作合成中的根本挑战。现有模型常假设地形平坦或无法通过文本控制动作语义。为此,我们提出SCENIC,一种扩散模型,可在虚拟场景中生成适应动态地形的人类动作,并通过自然语言实现语义控制。核心难点在于同时推理复杂场景几何与保持文本控制。模型需兼顾高层导航目标与细粒度环境约束,确保动作在不同地形上的物理合理性与精确导航,同时保留用户指定的文本意图,如‘小心跨过障碍’或‘像僵尸一样上楼梯’。我们的解决方案引入分层场景推理方法:核心是新颖的场景依赖、目标中心的规范化,辅以自我中心距离场捕捉局部几何细节。该双重表征使模型能在多样3D场景中生成物理合理动作。通过帧级文本对齐,系统实现不同动作风格间的平滑过渡,同时满足场景约束。实验表明,该扩散模型可生成任意长度的人类动作,既适应多变地形,又能响应文本提示。此外,我们验证了SCENIC在四个真实场景数据集上的泛化能力。代码、数据集与模型将公开于\url{https://virtualhumans.mpi-inf.mpg.de/scenic/}。

原文摘要 · Abstract (English)

Synthesizing natural human motion that adapts to complex environments while allowing creative control remains a fundamental challenge in motion synthesis. Existing models often fall short, either by assuming flat terrain or lacking the ability to control motion semantics through text. To address these limitations, we introduce SCENIC, a diffusion model designed to generate human motion that adapts to dynamic terrains within virtual scenes while enabling semantic control through natural language. The key technical challenge lies in simultaneously reasoning about complex scene geometry while maintaining text control. This requires understanding both high-level navigation goals and fine-grained environmental constraints. The model must ensure physical plausibility and precise navigation across varied terrain, while also preserving user-specified text control, such as ``carefully stepping over obstacles" or ``walking upstairs like a zombie." Our solution introduces a hierarchical scene reasoning approach. At its core is a novel scene-dependent, goal-centric canonicalization that handles high-level goal constraint, and is complemented by an ego-centric distance field that captures local geometric details. This dual representation enables our model to generate physically plausible motion across diverse 3D scenes. By implementing frame-wise text alignment, our system achieves seamless transitions between different motion styles while maintaining scene constraints. Experiments demonstrate our novel diffusion model generates arbitrarily long human motions that both adapt to complex scenes with varying terrain surfaces and respond to textual prompts. Additionally, we show SCENIC can generalize to four real-scene datasets. Our code, dataset, and models will be released at \url{https://virtualhumans.mpi-inf.mpg.de/scenic/}.

动作生成扩散模型语义控制虚拟人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。