用草图实时生成可编辑的逼真3D人脸,一步完成。
SketchFaceGS: Real-Time Sketch-Driven Face Editing and Generation with Gaussian Splatting

- 先粗后精的两阶段架构,从草图重建几何一致的纹理图
- 单次前向传播生成高保真3D人脸,支持实时自由视角修改
- 适合需要快速原型设计的数字人创作与交互应用
3D高斯表示已成为实现高质量数字头像建模的强大范式,可在实时渲染中达到逼真效果。然而,直观且交互式地创建或编辑3D高斯头像模型仍具挑战性。尽管2D草图提供了快速、直观的概念设计交互方式,但其信息稀疏、深度模糊且缺乏高频外观特征,难以从笔触中推断出密集且几何一致的3D高斯结构,尤其在实时约束下更为困难。为此,我们提出SketchFaceGS,首个基于草图的实时生成与编辑逼真3D高斯头像的框架。方法采用前馈式粗到精架构:基于Transformer的UV特征预测模块首先从输入草图重建粗略但几何一致的UV特征图;随后,3D UV特征增强模块引入高频、逼真的细节,生成高保真3D头像。对于编辑,我们提出UV掩码融合技术结合分层特征融合策略,实现精确、实时、多视角修改。大量实验表明,SketchFaceGS在生成保真度和编辑灵活性上均优于现有方法,能够在单次前向传播中从草图生成高质量、可编辑的3D头像。
原文摘要 · Abstract (English)
3D Gaussian representations have emerged as a powerful paradigm for digital head modeling, achieving photorealistic quality with real-time rendering. However, intuitive and interactive creation or editing of 3D Gaussian head models remains challenging. Although 2D sketches provide an ideal interaction modality for fast, intuitive conceptual design, they are sparse, depth-ambiguous, and lack high-frequency appearance cues, making it difficult to infer dense, geometrically consistent 3D Gaussian structures from strokes - especially under real-time constraints. To address these challenges, we propose SketchFaceGS, the first sketch-driven framework for real-time generation and editing of photorealistic 3D Gaussian head models from 2D sketches. Our method uses a feed-forward, coarse-to-fine architecture. A Transformer-based UV feature-prediction module first reconstructs a coarse but geometrically consistent UV feature map from the input sketch, and then a 3D UV feature enhancement module refines it with high-frequency, photorealistic detail to produce a high-fidelity 3D head. For editing, we introduce a UV Mask Fusion technique combined with a layer-by-layer feature-fusion strategy, enabling precise, real-time, free-viewpoint modifications. Extensive experiments show that SketchFaceGS outperforms existing methods in both generation fidelity and editing flexibility, producing high-quality, editable 3D heads from sketches in a single forward pass.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。