让3D训练的运动语言模型直接用2D动作输入，无需改动原模型。
A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models
- 设计可插拔2D动作接口，让3D预训练模型直接处理2D动作数据
- 在多个数据集上表现接近3D输入，优于从零训练的2D模型
- 实拍视频测试验证2D动作在真实场景下的实用性
运动语言模型（MoLM）通常通过将3D动作分词并用语言模型处理来理解人体运动。然而，从单目视频获取准确的3D动作仍具挑战性，限制了其在真实场景中的应用。为此，我们提出一种可插拔的2D动作接口，使预训练的3D MoLM能直接接收2D动作输入，无需修改或微调原模型。在公开数据集上的实验表明，该方法在多个MoLM上性能接近3D输入，并优于从零训练的2D MoLM。我们进一步构建了一个单目真实视频动作评估数据集，并引入真实视频适配器，证明在单目姿态估计设定下，2D动作相较于3D动作更具实用性。结果表明，2D动作为部署MoLM于真实世界运动理解场景提供了可行接口。代码已开源。
原文摘要 · Abstract (English)
Motion Language Models (MoLMs) typically understand human motions by tokenizing 3D motion and processing the resulting tokens using a language model. However, obtaining accurate 3D motions from monocular videos is challenging, limiting their real-world applicability. To address this issue, we introduce a plug-and-play 2D Motion Interface that enables 3D-pretrained MoLMs to accept 2D motion inputs without modifying or fine-tuning the original models. Experiments on public datasets show that our method achieves performance comparable to 3D motion inputs across multiple MoLMs and outperforms training MoLMs from scratch on 2D motions. We further construct a monocular real-world video motion evaluation dataset and introduce a real-video adapter, demonstrating the usefulness of 2D motions over 3D motions under the evaluated monocular pose-estimation setting. These results suggest that 2D motion provides a practical interface for deploying MoLMs in real-world motion understanding settings. Code is available at https://github.com/irajisamurai/2D-Motion-Interface.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。