联合重建多人、场景与相机，实现厘米级精度的环境定位。
Reconstructing People, Places, and Cameras
- 融合人体统计模型与传统SfM框架，同步优化人体网格、场景点云和相机参数。
- 在EgoHumans上人位姿误差从3.51米降至1.04米，相机姿态准确率提升20.3%。
- 适合需要精准人体-环境空间关系的自动驾驶、VR/AR研究者使用。
我们提出“从运动恢复人体与结构”（HSfM），一种从少量未标定多视角含人图像中，联合重建多个个体三维网格、场景点云及相机参数的方法，实现度量尺度的世界坐标系。该方法结合数据驱动的场景重建与传统运动结构（SfM）框架，在提升场景重建与相机估计精度的同时，恢复多人三维网格。相比现有缺乏度量尺度的方法，本方法通过人体统计模型估算近似度量尺度，并在同一世界坐标系下重建多人网格与场景点云，有效捕捉个体间及人与环境的空间关系。采用鲁棒基础模型初始化后，联合优化三者，协同提升各组件精度。在EgoHumans和EgoExo4D两个挑战性基准上对比验证，显著提升世界坐标系下的人体定位精度（EgoHumans误差由3.51米降至1.04米，EgoExo4D由2.9米降至0.56米）。此外，引入人体信息使相机姿态估计提升显著（如在EgoHumans上RRA@15提升20.3%）。定性结果也表明整体场景重建质量得到改善。代码已开源：https://github.com/hongsukchoi/HSfM_RELEASE
原文摘要 · Abstract (English)
We present "Humans and Structure from Motion" (HSfM), a method for jointly reconstructing multiple human meshes, scene point clouds, and camera parameters in a metric world coordinate system from a sparse set of uncalibrated multi-view images featuring people. Our approach combines data-driven scene reconstruction with the traditional Structure-from-Motion (SfM) framework to achieve more accurate scene reconstruction and camera estimation, while simultaneously recovering human meshes. In contrast to existing scene reconstruction and SfM methods that lack metric scale information, our method estimates approximate metric scale by leveraging a human statistical model. Furthermore, it reconstructs multiple human meshes within the same world coordinate system alongside the scene point cloud, effectively capturing spatial relationships among individuals and their positions in the environment. We initialize the reconstruction of humans, scenes, and cameras using robust foundational models and jointly optimize these elements. This joint optimization synergistically improves the accuracy of each component. We compare our method to existing approaches on two challenging benchmarks, EgoHumans and EgoExo4D, demonstrating significant improvements in human localization accuracy within the world coordinate frame (reducing error from 3.51m to 1.04m in EgoHumans and from 2.9m to 0.56m in EgoExo4D). Notably, our results show that incorporating human data into the SfM pipeline improves camera pose estimation (e.g., increasing RRA@15 by 20.3% on EgoHumans). Additionally, qualitative results show that our approach improves overall scene reconstruction quality. Our code is available at: https://github.com/hongsukchoi/HSfM_RELEASE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。