用触觉信息提升3D生成的几何细节真实度
Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation
- 融合高分辨率触觉法向图与视觉扩散模型，生成更真实的三维纹理
- 通过TextureDreambooth实现局部触觉纹理优化，提升细节精度
- 适用于需要精细触觉反馈的3D内容生成场景
现有3D生成方法依赖视觉扩散模型，常导致几何细节不足，表面过于平滑或细节错误地编码在反照率贴图中。为此，我们提出一种新方法，引入触觉作为额外模态以改善生成3D资产的几何细节。设计轻量级3D纹理场，联合合成视觉与触觉纹理，分别由视觉和触觉域的2D扩散模型先验引导。将视觉纹理生成条件化于高分辨率触觉法向图，并利用定制化的TextureDreambooth指导基于块的触觉纹理细化。进一步提出多部分生成流水线，支持不同区域的差异化纹理合成。据我们所知，这是首个利用高分辨率触觉传感增强3D生成几何细节的工作。在文本到3D和图像到3D设置下进行评估，实验表明本方法可在保持视觉与触觉模态准确对齐的同时，生成定制化且逼真的精细几何纹理。
原文摘要 · Abstract (English)
3D generation methods have shown visually compelling results powered by diffusion image priors. However, they often fail to produce realistic geometric details, resulting in overly smooth surfaces or geometric details inaccurately baked in albedo maps. To address this, we introduce a new method that incorporates touch as an additional modality to improve the geometric details of generated 3D assets. We design a lightweight 3D texture field to synthesize visual and tactile textures, guided by 2D diffusion model priors on both visual and tactile domains. We condition the visual texture generation on high-resolution tactile normals and guide the patch-based tactile texture refinement with a customized TextureDreambooth. We further present a multi-part generation pipeline that enables us to synthesize different textures across various regions. To our knowledge, we are the first to leverage high-resolution tactile sensing to enhance geometric details for 3D generation tasks. We evaluate our method in both text-to-3D and image-to-3D settings. Our experiments demonstrate that our method provides customized and realistic fine geometric textures while maintaining accurate alignment between two modalities of vision and touch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。