用语言模型统一生成与理解3D手物交互序列，支持文本和部分动作生成完整动作。
HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models
- 基于语言模型，通过新型分解式编码器将3D手物交互转为离散令牌。
- 文本输入可生成完整手物网格序列，部分序列可补全并生成描述文本。
- 在多个数据集上刷新生成与理解性能，适合动作生成与人机交互研究者。
我们提出HOIGPT，一种基于令牌的生成方法，统一了3D手物交互（HOI）的感知与生成，首次实现从多样化条件信号（如文本、物体、部分序列）中生成高质量3D HOI序列并进行描述。核心是利用大语言模型预测HOI序列与自然语言之间的双向转换。给定文本输入，HOIGPT生成手部与物体的网格序列；给定（部分）HOI序列，生成文本描述并完成序列。为使大语言模型更好地理解HOI，本文提出两项关键创新：(1) 一种新型物理合理的HOI分词器——手物分解的VQ-VAE，用于离散化HOI序列；(2) 一种可处理与生成文本及HOI令牌的运动感知语言模型。大量实验表明，HOIGPT在多个任务与基准上均达到新最优表现，文本生成提升2.01% R Precision，HOI生成降低2.56 FID。
原文摘要 · Abstract (English)
We introduce HOIGPT, a token-based generative method that unifies 3D hand-object interactions (HOI) perception and generation, offering the first comprehensive solution for captioning and generating high-quality 3D HOI sequences from a diverse range of conditional signals (\eg text, objects, partial sequences). At its core, HOIGPT utilizes a large language model to predict the bidrectional transformation between HOI sequences and natural language descriptions. Given text inputs, HOIGPT generates a sequence of hand and object meshes; given (partial) HOI sequences, HOIGPT generates text descriptions and completes the sequences. To facilitate HOI understanding with a large language model, this paper introduces two key innovations: (1) a novel physically grounded HOI tokenizer, the hand-object decomposed VQ-VAE, for discretizing HOI sequences, and (2) a motion-aware language model trained to process and generate both text and HOI tokens. Extensive experiments demonstrate that HOIGPT sets new state-of-the-art performance on both text generation (+2.01% R Precision) and HOI generation (-2.56 FID) across multiple tasks and benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。