用纯像素空间扩散模型实现单图三维重建，更简单也更清晰。
PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation
- 基于纯ViT的像素空间扩散模型，直接处理点云图块。
- 无需隐空间压缩，训练从零开始，性能超越复杂隐空间模型。
- 对透明等模糊区域更鲁棒，几何结构更锐利，适合工业重建场景。
当前最先进的单图像3D重建方法通常依赖复杂的混合架构和损失函数，或通过将几何信息压缩到隐空间来利用预训练的隐空间扩散模型。本文表明，此类架构开销和复杂损失设计并非必需。我们提出一种极简的像素空间扩散Transformer（PointDiT），基于普通ViT构建，直接作用于原始3D点图块，并由预训练DINOv3提取的图像标记条件驱动。与现有隐空间扩散方法不同，我们的扩散主干完全从头训练，无需点图块分词器。尽管结构简单，该方法在性能上超越复杂隐空间扩散模型，同时远比混合方法更简洁。尤其在透明物体等高度模糊区域表现更优，生成的几何结构更清晰。
原文摘要 · Abstract (English)
State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。