通过分步修复纹理，实现主体身份一致的3D/4D生成
Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
- 先追踪视频中需修改区域，再用主体驱动的2D修复模型逐步填充
- 在多视角图像上修复后重投影到3D，显著提升身份一致性
- 适合需要保持人物或物体特征一致性的个性化3D生成场景
当前3D/4D生成方法虽追求逼真、高效与美观，却常无法跨视角保持主体语义身份。针对单张或多张图像的个性化生成（即主体驱动生成）能更好保留主体特征，但该方向仍待深入。本文提出TIRE（Track, Inpaint, REsplat）方法，以现有3D生成模型输出的初始3D资产为输入，通过视频追踪识别需修改区域，利用主体驱动的2D图像修复模型逐步填充；最后将修复后的多视角图像重投影回3D并保持一致性。大量实验表明，本方法在3D/4D生成中显著优于现有最先进方法，在身份保持方面表现更优。
原文摘要 · Abstract (English)
Current 3D/4D generation methods are usually optimized for photorealism, efficiency, and aesthetics. However, they often fail to preserve the semantic identity of the subject across different viewpoints. Adapting generation methods with one or few images of a specific subject (also known as Personalization or Subject-driven generation) allows generating visual content that align with the identity of the subject. However, personalized 3D/4D generation is still largely underexplored. In this work, we introduce TIRE (Track, Inpaint, REsplat), a novel method for subject-driven 3D/4D generation. It takes an initial 3D asset produced by an existing 3D generative model as input and uses video tracking to identify the regions that need to be modified. Then, we adopt a subject-driven 2D inpainting model for progressively infilling the identified regions. Finally, we resplat the modified 2D multi-view observations back to 3D while still maintaining consistency. Extensive experiments demonstrate that our approach significantly improves identity preservation in 3D/4D generation compared to state-of-the-art methods. Our project website is available at https://zsh2000.github.io/track-inpaint-resplat.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。