用2D手绘线生成可动画3D模型,支持结构控制与高质量渲染。
Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion models
- 分两阶段生成:先根据手绘线和文字生成骨骼结构,再合成带纹理的网格。
- 生成的3D骨骼结构合理,网格质量高,骨架与模型对齐度好。
- 适合游戏、动画设计人员快速创建可动3D角色,无需专业建模技能。
可动画的3D资产是3D变形与动画的基础。现有3D生成方法难以生成可动画几何体,而传统绑定技术缺乏对骨骼结构的精细控制。为此,我们提出Stroke3D,一种直接从用户输入的2D手绘线与文本提示生成绑定网格的新框架。该方法采用两阶段流程:第一阶段为可控骨骼生成,利用骨骼图变分自编码器(Sk-VAE)将骨骼图结构编码至隐空间,通过骨骼图扩散模型(Sk-DiT)生成骨骼嵌入,结合文本语义与2D手绘线实现显式结构控制,再由VAE解码器重建高质量3D骨骼;第二阶段为增强网格合成,基于TextuRig数据集(从Objaverse-XL中整理的带描述的带纹理绑定网格)扩充训练数据,并采用基于骨架-网格对齐评分的偏好优化策略SKA-DPO,进一步提升几何保真度。实验表明,Stroke3D能生成合理骨架与高质量网格。据我们所知,这是首个基于用户手绘2D线条生成绑定3D网格的工作。
原文摘要 · Abstract (English)
Rigged 3D assets are fundamental to 3D deformation and animation. However, existing 3D generation methods face challenges in generating animatable geometry, while rigging techniques lack fine-grained structural control over skeleton creation. To address these limitations, we introduce Stroke3D, a novel framework that directly generates rigged meshes from user inputs: 2D drawn strokes and a descriptive text prompt. Our approach pioneers a two-stage pipeline that separates the generation into: 1) Controllable Skeleton Generation, we employ the Skeletal Graph VAE (Sk-VAE) to encode the skeleton's graph structure into a latent space, where the Skeletal Graph DiT (Sk-DiT) generates a skeletal embedding. The generation process is conditioned on both the text for semantics and the 2D strokes for explicit structural control, with the VAE's decoder reconstructing the final high-quality 3D skeleton; and 2) Enhanced Mesh Synthesis via TextuRig and SKA-DPO, where we then synthesize a textured mesh conditioned on the generated skeleton. For this stage, we first enhance an existing skeleton-to-mesh model by augmenting its training data with TextuRig: a dataset of textured and rigged meshes with captions, curated from Objaverse-XL. Additionally, we employ a preference optimization strategy, SKA-DPO, guided by a skeleton-mesh alignment score, to further improve geometric fidelity. Together, our framework enables a more intuitive workflow for creating ready to animate 3D content. To the best of our knowledge, our work is the first to generate rigged 3D meshes conditioned on user-drawn 2D strokes. Extensive experiments demonstrate that Stroke3D produces plausible skeletons and high-quality meshes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。