用Transformer高效重建手持物体3D点云，精度高且速度快。
HORT: Monocular Hand-held Objects Reconstruction with Transformers
- 基于粗到精策略，用图像特征逐步生成稠密点云。
- 在真实和合成数据上均达到当前最佳精度，推理速度更快。
- 融合手部几何与图像特征，提升物体姿态与形状重建效果。
从单目图像中重建手持物体的3D结构仍是计算机视觉中的重大挑战。现有方法多依赖隐式3D表示，导致结果过于平滑且生成显式3D形状耗时。近期基于扩散模型的方法虽可直接生成点云，但多步去噪过程使高分辨率重建效率低下。为此，我们提出一种基于Transformer的模型，可高效重建手持物体的稠密3D点云。方法采用粗到精策略：先从图像生成稀疏点云，再利用像素对齐的图像特征逐步细化为稠密表示。为提升精度，将图像特征与3D手部几何信息结合，联合预测物体点云及其相对于手部的姿态。模型端到端训练以实现最优性能。在合成与真实数据集上的实验表明，本方法在保持更高精度的同时显著提升推理速度，并能良好泛化至野外图像。
原文摘要 · Abstract (English)
Reconstructing hand-held objects in 3D from monocular images remains a significant challenge in computer vision. Most existing approaches rely on implicit 3D representations, which produce overly smooth reconstructions and are time-consuming to generate explicit 3D shapes. While more recent methods directly reconstruct point clouds with diffusion models, the multi-step denoising makes high-resolution reconstruction inefficient. To address these limitations, we propose a transformer-based model to efficiently reconstruct dense 3D point clouds of hand-held objects. Our method follows a coarse-to-fine strategy, first generating a sparse point cloud from the image and progressively refining it into a dense representation using pixel-aligned image features. To enhance reconstruction accuracy, we integrate image features with 3D hand geometry to jointly predict the object point cloud and its pose relative to the hand. Our model is trained end-to-end for optimal performance. Experimental results on both synthetic and real datasets demonstrate that our method achieves state-of-the-art accuracy with much faster inference speed, while generalizing well to in-the-wild images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。