用像素快速预测3D场景物理属性，支持真实世界零样本泛化。
Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels
- 基于监督学习直接从视觉特征预测材料属性，无需逐场景优化。
- 推理速度比传统方法快1.46至4.39倍，且在多个场景上表现稳定。
- 可零样本迁移至真实图像，适合虚拟现实与物理仿真应用。
从视觉信息中推断3D场景的物理属性是构建交互式、逼真虚拟世界的关键挑战。尽管人类能直观理解弹性或刚度等材质特性，现有方法通常依赖缓慢的逐场景优化，限制了泛化能力与应用范围。为此，我们提出PIXIE，一种新型方法，通过纯监督损失训练通用神经网络，从3D视觉特征中预测跨多场景的物理属性。训练完成后，其前馈网络可快速生成合理的材料场分布，结合如高斯泼溅（Gaussian Splatting）等学习到的静态场景表示，实现外部力作用下的真实物理模拟。为推动研究，我们还构建了PIXIEVERSE，目前最大的配对3D资产与物理材料标注数据集之一。大量评估表明，PIXIE相比测试时优化方法性能提升1.46–4.39倍，且速度达数量级提升。借助CLIP等预训练视觉特征，该方法即使仅在合成数据上训练，也能实现对真实场景的零样本泛化。
原文摘要 · Abstract (English)
Inferring the physical properties of 3D scenes from visual information is a critical yet challenging task for creating interactive and realistic virtual worlds. While humans intuitively grasp material characteristics such as elasticity or stiffness, existing methods often rely on slow, per-scene optimization, limiting their generalizability and application. To address this problem, we introduce PIXIE, a novel method that trains a generalizable neural network to predict physical properties across multiple scenes from 3D visual features purely using supervised losses. Once trained, our feed-forward network can perform fast inference of plausible material fields, which coupled with a learned static scene representation like Gaussian Splatting enables realistic physics simulation under external forces. To facilitate this research, we also collected PIXIEVERSE, one of the largest known datasets of paired 3D assets and physic material annotations. Extensive evaluations demonstrate that PIXIE is about 1.46-4.39x better and orders of magnitude faster than test-time optimization methods. By leveraging pretrained visual features like CLIP, our method can also zero-shot generalize to real-world scenes despite only ever been trained on synthetic data. https://pixie-3d.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。