让文字生成的肢体动作自然融入场景,不牺牲多样性。
SceneAdapt: Scene-aware Adaptation of Human Motion Diffusion
- 用动作插值作为桥梁,连接文本与场景数据
- 在不依赖大量配对数据下实现场景感知生成
- 适合需要真实场景交互的动作生成研究者
人体动作具有丰富的语义和多样性,同时受周围环境影响。现有生成方法难以在保持语义多样性的同时满足几何场景约束,因构建兼具丰富文本-动作覆盖和精确场景交互的大规模数据集极为困难。本文提出SceneAdapt,一种两阶段自适应框架,可在无大规模文本-场景-动作配对数据条件下,实现从文本生成语义多样且场景感知的人体动作。核心思路是利用无需文本的动作插值(motion inbetweening)作为桥梁,连接文本-动作数据集与场景-动作数据集。通过先进行动作插值自适应,再进行场景感知插值自适应,将几何场景约束注入文本条件生成过程,同时保持语义多样性。为支持插值适应,提出上下文感知关键帧层(CaKey),在关键帧条件合成中调节动作隐变量并保留原始隐空间结构;为支持场景感知插值,引入场景条件层(SceneCo),通过交叉注意力自适应查询局部上下文以注入几何场景信息。实验表明,SceneAdapt有效将场景感知能力注入文本到动作模型,且不损失语义多样性,并进一步分析了该感知机制的形成路径。代码与模型将公开。项目页:https://sceneadapt.github.io/
原文摘要 · Abstract (English)
Human motion is inherently diverse and semantically rich, while also shaped by the surrounding scene. However, existing motion generation approaches fail to generate semantically diverse motion while simultaneously respecting geometric scene constraints, since constructing large-scale datasets with both rich text-motion coverage and precise scene interactions is extremely challenging. In this work, we introduce SceneAdapt, a two-stage adaptation framework that enables semantically diverse, scene-aware human motion generation from text without large-scale paired text--scene--motion data. Our key idea is to use motion inbetweening, a learnable proxy task that requires no text, as a bridge between two disjoint resources: a text-motion dataset and a scene-motion dataset. By first adapting a text-to-motion model through inbetweening and then through scene-aware inbetweening, SceneAdapt injects geometric scene constraints into text-conditioned generation while preserving semantic diversity. To enable adaptation for inbetweening, we propose a novel Context-aware Keyframing (CaKey) layer that modulates motion latents for keyframe-conditioned synthesis while preserving the original latent manifold. To further adapt the model for scene-aware inbetweening, we introduce a Scene-conditioning (SceneCo) layer that injects geometric scene information by adaptively querying local context via cross-attention. Experimental results show that SceneAdapt effectively injects scene-awareness into text-to-motion models without sacrificing semantic diversity, and we further analyze the mechanisms through which this awareness emerges. Code and models will be released. Project page: \href{https://sceneadapt.github.io/}{sceneadapt.github.io}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。