arXiv:2504.17441cs.CV2025-04ICCV被引 10

通过自迭代循环提升4D物体理解,让模型越看越准。

Predict-Optimize-Distill: A Self-Improving Cycle for 4D Object Understanding

  • 用预测-优化-蒸馏循环,边看边改进模型与运动轨迹。
  • 在14个真实物体上比纯优化方法提升显著,视频越长效果越好。
  • 适合需要长期观察、复杂运动的物体建模任务。

人类可通过长时间观察来建立对未见物体3D形态的直觉,观察时间越长,预测其3D状态的能力越强。现有系统或从多视角观测中优化底层表示,或基于监督数据训练前馈预测器。本文提出预测-优化-蒸馏(POD)框架,通过预测与优化的相互增强循环,实现更优的4D物体理解。给定多视角物体扫描和人类-物体交互的长时单目视频,POD迭代训练神经网络从RGB帧预测局部部件姿态,利用该预测初始化全局优化,通过逆渲染精修姿态输出,最后将优化结果转化为新视角的合成自标注数据回传给模型进行蒸馏。每轮迭代同时提升模型预测能力与优化轨迹质量,形成自我增强的训练闭环,逐步学习物体姿态配置。此外,引入准多视角挖掘策略以缓解深度模糊问题。我们在14个真实物体和5个合成物体上评估,涵盖旋转、滑动关节及独立分离/重组的多体结构。实验表明,相比陷入局部最优的纯优化基线,POD性能显著提升,尤其在长视频条件下;且性能随视频长度与迭代次数增长,体现其随观测增加而可扩展的优化能力。

原文摘要 · Abstract (English)

Humans can resort to long-form inspection to build intuition on predicting the 3D configurations of unseen objects. The more we observe the object motion, the better we get at predicting its 3D state immediately. Existing systems either optimize underlying representations from multi-view observations or train a feed-forward predictor from supervised datasets. We introduce Predict-Optimize-Distill (POD), a self-improving framework that interleaves prediction and optimization in a mutually reinforcing cycle to achieve better 4D object understanding with increasing observation time. Given a multi-view object scan and a long-form monocular video of human-object interaction, POD iteratively trains a neural network to predict local part poses from RGB frames, uses this predictor to initialize a global optimization which refines output poses through inverse rendering, then finally distills the results of optimization back into the model by generating synthetic self-labeled training data from novel viewpoints. Each iteration improves both the predictive model and the optimized motion trajectory, creating a virtuous cycle that bootstraps its own training data to learn about the pose configurations of an object. We also introduce a quasi-multiview mining strategy for reducing depth ambiguity by leveraging long video. We evaluate POD on 14 real-world and 5 synthetic objects with various joint types, including revolute and prismatic joints as well as multi-body configurations where parts detach or reattach independently. POD demonstrates significant improvement over a pure optimization baseline which gets stuck in local minima, particularly for longer videos. We also find that POD's performance improves with both video length and successive iterations of the self-improving cycle, highlighting its ability to scale performance with additional observations and looped refinement.

4D理解自迭代姿态估计逆渲染

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。