单图生成高保真3D网格，通过可插拔变形模块提升一致性与清晰度。
Fancy123: One Image to High-Quality 3D Mesh Generation via Plug-and-Play Deformation
- 用可插拔变形模块修复多视角图像对齐问题
- 使生成网格更贴合输入图像，减少模糊现象
- 适用于多种现有单图转3D方法，部署灵活
从单张图像生成3D网格是一个重要但病态的问题。现有方法主要依赖2D多视角扩散模型生成中间多视角图像，再通过大重建模型（LRM）生成最终网格。然而，多视角图像存在局部不一致，且生成网格常缺乏对输入图像的保真度或显得模糊。我们提出Fancy123，包含两个增强模块和一个反投影操作，分别解决上述三类问题：外观增强模块对2D多视角图像进行形变，重对齐错位像素以提升多视角一致性；保真度增强模块对3D网格进行形变，使其更匹配输入图像；将输入图像与形变后的多视角图像反投影至LRM生成的网格上，确保高清晰度，摒弃了LRM预测的模糊颜色。大量定性与定量实验验证了Fancy123在性能上达到当前最优水平，显著提升。此外，两个增强模块为即插即用设计，可在推理阶段无缝集成到多种现有单图转3D方法中。代码已公开于：https://github.com/YuQiao0303/Fancy123
原文摘要 · Abstract (English)
Generating 3D meshes from a single image is an important but ill-posed task. Existing methods mainly adopt 2D multiview diffusion models to generate intermediate multiview images, and use the Large Reconstruction Model (LRM) to create the final meshes. However, the multiview images exhibit local inconsistencies, and the meshes often lack fidelity to the input image or look blurry. We propose Fancy123, featuring two enhancement modules and an unprojection operation to address the above three issues, respectively. The appearance enhancement module deforms the 2D multiview images to realign misaligned pixels for better multiview consistency. The fidelity enhancement module deforms the 3D mesh to match the input image. The unprojection of the input image and deformed multiview images onto LRM's generated mesh ensures high clarity, discarding LRM's predicted blurry-looking mesh colors. Extensive qualitative and quantitative experiments verify Fancy123's SoTA performance with significant improvement. Also, the two enhancement modules are plug-and-play and work at inference time, allowing seamless integration into various existing single-image-to-3D methods. Code at: https://github.com/YuQiao0303/Fancy123
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。