用分块掩码建模生成逼真3D人互动,效果优于现有方法。
InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling
- 将动作转为2D离散符号图,保留时空细节并增强空间感知
- 掩码预测训练使双人互动生成更真实,FID低至5.154(InterHuman)
- 无需重训即可生成反应动作,适合动画与虚拟交互场景
从文本描述生成逼真的3D人与人互动仍是难题。现有基于扩散模型的方法常缺乏真实感与保真度。本文提出InterMask,一种在离散空间中通过协作掩码建模生成人互动的新框架。首先利用VQ-VAE将每个动作序列转化为2D离散动作符号图,相比传统1D符号图,更好保留精细时空细节并提升空间感知。在此表示基础上,采用专为捕捉复杂时空相互依赖设计的Transformer架构,协同建模两人动作符号。训练时随机掩码双方动作符号并学习重建;推理时从全掩码序列开始逐步填充。凭借增强的动作表征、专用架构与有效学习策略,InterMask达到当前最优性能:在InterHuman数据集上FID为5.154(优于in2IN的5.535),在InterX数据集上FID为0.399(远优于InterGen的5.207)。此外,无需重新设计或微调即可自然生成反应动作。
原文摘要 · Abstract (English)
Generating realistic 3D human-human interactions from textual descriptions remains a challenging task. Existing approaches, typically based on diffusion models, often produce results lacking realism and fidelity. In this work, we introduce InterMask, a novel framework for generating human interactions using collaborative masked modeling in discrete space. InterMask first employs a VQ-VAE to transform each motion sequence into a 2D discrete motion token map. Unlike traditional 1D VQ token maps, it better preserves fine-grained spatio-temporal details and promotes spatial awareness within each token. Building on this representation, InterMask utilizes a generative masked modeling framework to collaboratively model the tokens of two interacting individuals. This is achieved by employing a transformer architecture specifically designed to capture complex spatio-temporal inter-dependencies. During training, it randomly masks the motion tokens of both individuals and learns to predict them. For inference, starting from fully masked sequences, it progressively fills in the tokens for both individuals. With its enhanced motion representation, dedicated architecture, and effective learning strategy, InterMask achieves state-of-the-art results, producing high-fidelity and diverse human interactions. It outperforms previous methods, achieving an FID of $5.154$ (vs $5.535$ of in2IN) on the InterHuman dataset and $0.399$ (vs $5.207$ of InterGen) on the InterX dataset. Additionally, InterMask seamlessly supports reaction generation without the need for model redesign or fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。