不训练直接用DUSt3R注意力图，实现动态场景4D重建
Easi3R: Estimating Disentangled Motion from DUSt3R Without Training
- 推理时通过注意力自适应解耦运动信息
- 在真实动态视频上优于需训练的现有方法
- 适合无标注动态数据、追求轻量部署的研究者
DUSt3R最近实现了静态场景稠密点云和相机参数的鲁棒估计，依赖Transformer架构与大规模3D数据的直接监督。然而，可用4D数据集规模小、多样性差，严重制约了通用4D模型的训练。传统方法通常在可扩展的动态视频数据上微调3D模型，并引入光流、深度等几何先验。本文提出Easi3R，一种无需训练的高效4D重建方法：在推理阶段对注意力层进行自适应调整，避免从头预训练或网络微调。我们发现，DUSt3R中的注意力层天然编码了相机与物体运动信息。通过精细解耦这些注意力图，实现了准确的动态区域分割、相机位姿估计与4D稠密点图重建。在真实动态视频上的大量实验表明，该轻量级注意力自适应显著优于此前需在大量动态数据上训练或微调的先进方法。代码已公开，供研究使用。
原文摘要 · Abstract (English)
Recent advances in DUSt3R have enabled robust estimation of dense point clouds and camera parameters of static scenes, leveraging Transformer network architectures and direct supervision on large-scale 3D datasets. In contrast, the limited scale and diversity of available 4D datasets present a major bottleneck for training a highly generalizable 4D model. This constraint has driven conventional 4D methods to fine-tune 3D models on scalable dynamic video data with additional geometric priors such as optical flow and depths. In this work, we take an opposite path and introduce Easi3R, a simple yet efficient training-free method for 4D reconstruction. Our approach applies attention adaptation during inference, eliminating the need for from-scratch pre-training or network fine-tuning. We find that the attention layers in DUSt3R inherently encode rich information about camera and object motion. By carefully disentangling these attention maps, we achieve accurate dynamic region segmentation, camera pose estimation, and 4D dense point map reconstruction. Extensive experiments on real-world dynamic videos demonstrate that our lightweight attention adaptation significantly outperforms previous state-of-the-art methods that are trained or finetuned on extensive dynamic datasets. Our code is publicly available for research purpose at https://easi3r.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。