arXiv:2604.07273cs.CV2026-04被引 1

用真实视频训练3D扩散模型,生成逼真可动的全身虚拟形象。

GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos

  • 从部分可见的2D视频中构建3D令牌,实现大规模数据训练。
  • 通过可见性感知训练策略,解决视频中身体遮挡导致的模糊问题。
  • 支持文本/图像输入生成与编辑,适合影视动画与虚拟人开发。

我们提出GenLCA,一种基于扩散模型的生成方法,可从文本和图像输入生成并编辑逼真的全身虚拟形象。生成结果忠实于输入,同时支持高保真面部与全身动画。核心思想是提出一种新范式,仅需部分观测的2D数据即可训练全身体积扩散模型,使训练数据集扩展至百万级真实视频。为实现这一目标,我们利用预训练的前馈式虚拟形象重建模型作为可动画3D编码器,将非结构化视频帧转换为结构化3D令牌。然而,多数真实视频仅提供部分身体信息,导致3D令牌出现过度模糊或透明伪影。为此,我们提出一种可见性感知的扩散训练策略:将无效区域替换为可学习令牌,并仅在有效区域计算损失。随后,在该令牌数据集上训练流模型,从而保留预训练模型提供的保真度与可动画性。该方法成功实现以大规模真实视频数据在原生3D空间中训练扩散模型。实验表明,本方法在多样性和保真度上显著优于现有方案。

原文摘要 · Abstract (English)

We present GenLCA, a diffusion-based generative model for generating and editing photorealistic full-body avatars from text and image inputs. The generated avatars are faithful to the inputs, while supporting high-fidelity facial and full-body animations. The core idea is a novel paradigm that enables training a full-body 3D diffusion model from partially observable 2D data, allowing the training dataset to scale to millions of real-world videos. This scalability contributes to the superior photorealism and generalizability of GenLCA. Specifically, we scale up the dataset by repurposing a pretrained feed-forward avatar reconstruction model as an animatable 3D tokenizer, which encodes unstructured video frames into structured 3D tokens. However, most real-world videos only provide partial observations of body parts, resulting in excessive blurring or transparency artifacts in the 3D tokens. To address this, we propose a novel visibility-aware diffusion training strategy that replaces invalid regions with learnable tokens and computes losses only over valid regions. We then train a flow-based diffusion model on the token dataset, inherently maintaining the photorealism and animatability provided by the pretrained avatar reconstruction model. Our approach effectively enables the use of large-scale real-world video data to train a diffusion model natively in 3D. We demonstrate the efficacy of our method through diverse and high-fidelity generation and editing results, outperforming existing solutions by a large margin. The project page is available at https://onethousandwu.com/GenLCA-Page.

3D生成扩散模型虚拟人视频驱动

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。