用结构感知检索提升文本生成动作的准确性和速度。
ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation

- 设计分层对比学习与时空注意力融合动作特征,增强局部细节建模。
- 在KIT-ML和SnapMoGen上实现最低FID值,单阶段推理速度最快。
- 直接在生成器潜空间检索,解决语义与潜空间不匹配问题。
文本到动作(T2M)生成将自然语言映射为人体关节运动,助力游戏、虚拟现实和机器人应用。检索增强型文本到动作(RAG-T2M)通过条件化于检索到的动作-文本对,提升复杂描述下的生成质量。然而现有RAG-T2M模型存在两大挑战:粗粒度检索与融合机制忽略人体动作的层次化时空拓扑结构;且检索证据位于语义空间,与生成器潜空间存在表征鸿沟。为此,我们提出ReMoMask,通过分层双向动量对比学习对齐全局与局部特征,结合时空感知注意力实现拓扑融合,并引入拓扑结构掩码强制局部语义锚定。为解决第二点,提出ReMoMask-2,将检索数据库重建于生成器预量化潜空间中,通过轻量级投影器对齐文本查询,使生成器可直接消费检索动作的语义内容。在HumanML3D、KIT-ML和SnapMoGen上的实验表明,该方法检索精度达当前最优,ReMoMask-2在KIT-ML和SnapMoGen上取得最低FID,其单掩码变换器阶段性能超越原双阶段管道,推理速度最快。
原文摘要 · Abstract (English)
Text-to-motion (T2M) generation maps natural language to human joint movements, aiding gaming, VR, and robotics. Retrieval-Augmented Text-to-Motion (RAG-T2M) improves generation on complex descriptions by conditioning on retrieved motion-text pairs. However, existing RAG-T2M models face two challenges: coarse-grained retrieval and fusion mechanisms overlook the hierarchical, spatial-temporal topology of human motion, and a representation gap exists because retrieved evidence resides in a semantic space separate from the generator's latents. To address the first, we present ReMoMask, a structure-aware RAG framework coupling Hierarchical Bidirectional Momentum (HBM) contrastive learning to align global and part-level features with text; Semantic Spatial-Temporal Attention (SSTA) for topology-aware fusion; and Topology Structured Masking (TSM) to force robust part-level grounding via adaptive masking. To address the second, we introduce ReMoMask-2, which rebuilds the retrieval database directly within the generator's pre-quantization latent space and aligns text queries via a distilled lightweight projector, allowing the generator to directly consume the retrieved motion's semantic content. Extensive experiments on HumanML3D, KIT-ML, and SnapMoGen demonstrate our retriever achieves state-of-the-art accuracy, while ReMoMask-2 attains the lowest FID on KIT-ML and SnapMoGen; notably, its single mask-transformer stage surpasses ReMoMask's full two-stage pipeline and delivers the fastest inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。