用空中手势控制虚拟世界中手部交互,生成真实感视频。
Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures
- 通过投影3D手部网格实现遮挡无关的手部控制。
- 支持任意长度视频生成,提升3D一致性与画面稳定性。
- 适合增强现实与具身智能中的实时交互场景。
眼球视角交互模型对增强现实和具身智能至关重要,要求视觉生成能低延迟响应用户输入,保持几何一致性和长期稳定性。本文研究从单张场景图出发,基于自由空间手势生成眼球视角交互视频,目标是合成逼真视频:手部进入场景,与物体互动,并在头部运动下引发合理世界动态。该任务面临三大挑战:自由手势与接触密集训练数据之间的分布差异、单目视角下手部运动与相机运动的混淆,以及任意长度视频生成需求。我们提出Hand2World,一个统一的自回归框架:采用投影3D手部网格实现遮挡无关的手部条件,从场景上下文推断可见性与遮挡;通过逐像素普吕克射线嵌入显式注入相机几何信息,解耦相机运动与手部运动,防止背景漂移;并构建全自动单目标注流程,将双向扩散模型蒸馏为因果生成器,实现任意长度合成。在三个眼球视角交互基准上实验表明,该方法在感知质量与3D一致性上显著提升,支持相机控制与长时交互生成。
原文摘要 · Abstract (English)
Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability. We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion. This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation. We present Hand2World, a unified autoregressive framework that addresses these challenges through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal. To stabilize egocentric viewpoint changes, we inject explicit camera geometry via per-pixel Plücker-ray embeddings, disentangling camera motion from hand motion and preventing background drift. We further develop a fully automated monocular annotation pipeline and distill a bidirectional diffusion model into a causal generator, enabling arbitrary-length synthesis. Experiments on three egocentric interaction benchmarks show substantial improvements in perceptual quality and 3D consistency while supporting camera control and long-horizon interactive generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。