用自回归视频生成技术实现实时手物交互模拟,支持各种复杂物体。
SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation
- 基于因果推理的自回归框架,输入图像和手部动作流生成交互视频。
- 1.3B模型在RTX 5090上达18 FPS、150ms延迟,可生成超一分钟连续视频。
- 适用于虚拟试穿、游戏交互等需真实感手物交互的场景。
建模与合成复杂的手物交互仍是重大挑战,即使最先进的物理引擎也难以应对。传统基于仿真的方法依赖显式刚体模型和预设手势,无法有效捕捉与非刚性或关节结构物体(如柔性织物、弹性材料、铰链结构、毛绒表面甚至活体)的动态交互。本文提出SpriteHand,一种自回归视频生成框架,可实时合成涵盖多种物体类型与运动模式的手物交互视频。该模型输入静态物体图像及含手部动作的视频流,实时生成嵌入真实场景的虚拟物体交互效果。采用因果推理架构实现自回归生成,并结合混合后训练策略提升视觉真实感与时间连贯性。1.3B参数模型在单张NVIDIA RTX 5090 GPU上实现约18 FPS、640x368分辨率下的实时生成,延迟约150ms,支持超过一分钟的连续输出。实验表明,其在视觉质量、物理合理性与交互保真度方面均优于生成式与引擎基线方法。
原文摘要 · Abstract (English)
Modeling and synthesizing complex hand-object interactions remains a significant challenge, even for state-of-the-art physics engines. Conventional simulation-based approaches rely on explicitly defined rigid object models and pre-scripted hand gestures, making them inadequate for capturing dynamic interactions with non-rigid or articulated entities such as deformable fabrics, elastic materials, hinge-based structures, furry surfaces, or even living creatures. In this paper, we present SpriteHand, an autoregressive video generation framework for real-time synthesis of versatile hand-object interaction videos across a wide range of object types and motion patterns. SpriteHand takes as input a static object image and a video stream in which the hands are imagined to interact with the virtual object embedded in a real-world scene, and generates corresponding hand-object interaction effects in real time. Our model employs a causal inference architecture for autoregressive generation and leverages a hybrid post-training approach to enhance visual realism and temporal coherence. Our 1.3B model supports real-time streaming generation at around 18 FPS and 640x368 resolution, with an approximate 150 ms latency on a single NVIDIA RTX 5090 GPU, and more than a minute of continuous output. Experiments demonstrate superior visual quality, physical plausibility, and interaction fidelity compared to both generative and engine-based baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。