拖拽即可生成真实物理运动的3D模型,实时且无需重训练。
DragMesh: Interactive 3D Generation Made Easy
- 分离语义推理与几何回归,精准预测关节类型和位置。
- 用双四元数变分自编码器生成连续无奇点的运动轨迹。
- 支持新物体实时交互,适合需要快速原型设计的场景。
尽管生成模型在静态3D内容生成上表现优异,但实现理解物体运动与交互的系统仍是根本挑战。现有方法要么物理一致但速度慢,要么生成快却违背基本运动约束。我们提出DragMesh,一个基于轻量级运动生成核心的实时交互式3D可动性框架。核心创新是解耦的运动学推理与生成框架:首先通过我们的运动学预测网络(KPP-Net)将语义意图推理(决定关节类型)与几何回归(确定轴向与原点)分离;其次为利用双四元数表示刚体运动的紧凑、连续、无奇点特性,我们设计了新型双四元数变分自编码器(DQ-VAE)。该DQ-VAE接收预测的关节先验与用户拖拽输入,生成完整且合理的运动轨迹。为确保严格遵循运动学约束,我们在DQ-VAE非自回归Transformer解码器每一层注入关节先验,使用FiLM特征调制条件;辅以数值稳定的叉积损失,保证轴线对齐。此解耦设计使DragMesh实现实时性能,并可在不重新训练的情况下对新物体进行合理生成式操作,迈向生成式3D智能的重要一步。代码:https://github.com/AIGeeksGroup/DragMesh。网站:https://aigeeksgroup.github.io/DragMesh。
原文摘要 · Abstract (English)
While generative models have excelled at creating static 3D content, the pursuit of systems that understand how objects move and respond to interactions remains a fundamental challenge. Current methods for articulated motion lie at a crossroads: they are either physically consistent but too slow for real-time use, or generative but violate basic kinematic constraints. We present DragMesh, a robust framework for real-time interactive 3D articulation built around a lightweight motion generation core. Our core contribution is a novel decoupled kinematic reasoning and motion generation framework. First, we infer the latent joint parameters by decoupling semantic intent reasoning (which determines the joint type) from geometric regression (which determines the axis and origin using our Kinematics Prediction Network (KPP-Net)). Second, to leverage the compact, continuous, and singularity-free properties of dual quaternions for representing rigid body motion, we develop a novel Dual Quaternion VAE (DQ-VAE). This DQ-VAE receives these predicted priors, along with the original user drag, to generate a complete, plausible motion trajectory. To ensure strict adherence to kinematics, we inject the joint priors at every layer of the DQ-VAE's non-autoregressive Transformer decoder using FiLM (Feature-wise Linear Modulation) conditioning. This persistent, multi-scale guidance is complemented by a numerically-stable cross-product loss to guarantee axis alignment. This decoupled design allows DragMesh to achieve real-time performance and enables plausible, generative articulation on novel objects without retraining, offering a practical step toward generative 3D intelligence. Code: https://github.com/AIGeeksGroup/DragMesh. Website: https://aigeeksgroup.github.io/DragMesh.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。