构建首个跨模态人类运动预训练模型，支持轨迹与姿态预测。
Multi-Transmotion: Pre-trained Model for Human Motion Prediction
- 融合7个数据集，统一格式实现多模态预训练
- 在NBA、JTA轨迹和AMASS、3DPW姿态任务上表现优异
- 适合自动驾驶与社交机器人场景的运动预测研究者
智能系统预测人类行为的能力对自动驾驶导航与社会机器人等应用至关重要。然而，人类运动的复杂性导致缺乏标准化的人类运动预测数据集，制约了预训练模型的发展。本文通过整合多个数据集（涵盖轨迹与3D姿态关键点），构建统一格式，提出多模态预训练模型Multi-Transmotion。该模型基于Transformer架构，设计新型掩码策略以捕捉丰富表征。在多个下游任务中表现良好：在NBA与JTA数据集上进行轨迹预测，在AMASS与3DPW数据集上进行姿态预测。代码已开源：https://github.com/vita-epfl/multi-transmotion。
原文摘要 · Abstract (English)
The ability of intelligent systems to predict human behaviors is crucial, particularly in fields such as autonomous vehicle navigation and social robotics. However, the complexity of human motion have prevented the development of a standardized dataset for human motion prediction, thereby hindering the establishment of pre-trained models. In this paper, we address these limitations by integrating multiple datasets, encompassing both trajectory and 3D pose keypoints, to propose a pre-trained model for human motion prediction. We merge seven distinct datasets across varying modalities and standardize their formats. To facilitate multimodal pre-training, we introduce Multi-Transmotion, an innovative transformer-based model designed for cross-modality pre-training. Additionally, we present a novel masking strategy to capture rich representations. Our methodology demonstrates competitive performance across various datasets on several downstream tasks, including trajectory prediction in the NBA and JTA datasets, as well as pose prediction in the AMASS and 3DPW datasets. The code is publicly available: https://github.com/vita-epfl/multi-transmotion
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。