用自然语言高效个性化3D内容,无需重新训练。
Align 3D Representation and Text Embedding for 3D Content Personalization
- 设计相机条件的3D到文本逆映射,对齐3D与文本嵌入空间。
- 通过自然语言提示实现3D内容修改,效果媲美重训练方法。
- 适合需要快速定制3D资产的设计师与开发者使用。
NeRF和3DGS等技术显著提升了3D内容生成的效率与质量,但高效个性化仍面临挑战。现有方法多依赖知识蒸馏,需耗时重训练。为此,我们提出Invert3D框架,通过建立3D表示与文本嵌入空间的对齐,实现便捷个性化。该方法设计了相机条件的3D-to-text逆映射,将3D内容投影至与文本嵌入对齐的3D嵌入空间。由此,用户可直接通过自然语言提示高效操控和个性化3D内容,无需重新训练。大量实验表明,Invert3D能有效实现3D内容个性化。代码已开源:https://github.com/qsong2001/Invert3D。
原文摘要 · Abstract (English)
Recent advances in NeRF and 3DGS have significantly enhanced the efficiency and quality of 3D content synthesis. However, efficient personalization of generated 3D content remains a critical challenge. Current 3D personalization approaches predominantly rely on knowledge distillation-based methods, which require computationally expensive retraining procedures. To address this challenge, we propose \textbf{Invert3D}, a novel framework for convenient 3D content personalization. Nowadays, vision-language models such as CLIP enable direct image personalization through aligned vision-text embedding spaces. However, the inherent structural differences between 3D content and 2D images preclude direct application of these techniques to 3D personalization. Our approach bridges this gap by establishing alignment between 3D representations and text embedding spaces. Specifically, we develop a camera-conditioned 3D-to-text inverse mechanism that projects 3D contents into a 3D embedding aligned with text embeddings. This alignment enables efficient manipulation and personalization of 3D content through natural language prompts, eliminating the need for computationally retraining procedures. Extensive experiments demonstrate that Invert3D achieves effective personalization of 3D content. Our work is available at: https://github.com/qsong2001/Invert3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。