arXiv:2609.08250cs.ROcs.AI2026-09

提出新测试方法,区分视觉模型是否真懂物理规律。

CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations

论文配图:CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations
图 1 · 摘自论文原文
  • 用两次已知撞击+一次接触瞬间画面,让模型推断物体滑行距离。
  • 固定场景下所有模型表现接近满分,但换场景后差距达0.50 R²。
  • 适合评估机器人感知模型的物理理解能力,尤其关注真实环境适应性。

物体滑动距离取决于质量与摩擦力,单张图像无法揭示这些信息。当前预训练视觉编码器常被用于操控任务的世界模型感知前端,其物理推理能力通常通过扰动基准和线性探测器在干净、固定视角场景中评估。我们发现这类评估无法区分真正理解物理的模型与仅依赖表面特征的模型。为此提出CALIPER(校准后预测):一个未知质量与摩擦的物体被以已知速度撞击两次,第三次只展示至接触瞬间,线性读出需预测滑行距离;若替换为另一物体的校准片段,预测应失效。在2000次模拟实验与8种表征(从V-JEPA 2到随机初始化ViT及原始像素)中,校准带来+0.50 R²提升,互换校准片段则完全抵消该收益。但在固定摄像头场景中,所有表征均接近0.02 R²内最优表现,因位移直接反映在像素坐标上。当相机、光照、遮挡动态变化时,同一批表征的性能跨度达0.50 R²;此时,尽管使用真实状态作为参考,V-JEPA 2误差仅4毫米,而随机初始化的ViT误差达20毫米,甚至不如忽略物体。线性探测器对此毫无敏感性:帧聚合方式改变的影响超过预训练差异,且在特定场景中抹除质量方向信息几乎无损,而在另一场景中损失0.35 R²。能否对模型进行有效排序取决于具体评测设置,本文提供三项实证检验以确立评估有效性。

原文摘要 · Abstract (English)

How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are increasingly used as the perception front end of world models for manipulation, and their physical competence is assessed with perturbation benchmarks and linear probes, almost always in a clean, fixed-camera scene. We show that these assessments cannot distinguish an encoder that infers physics from one that does not. CALIPER (calibrate, then predict) is a direct test: an object of unknown mass and friction is struck twice at known speeds, a third strike is shown only up to the moment of contact, and a linear readout on frozen features must predict how far the object slides. Swapping in another object's calibration clips checks that the evidence is actually used. Across 2,000 simulated episodes and eight representations, from V-JEPA 2 to a randomly initialised ViT and raw pixels, calibration adds +0.50 R^2 and the swap removes it. Yet in the clean scene every representation lands within 0.02 R^2 of the ceiling set by true simulator state, because a fixed camera exposes the object's displacement directly in pixel coordinates. Resampling camera, lighting, and clutter for every clip spreads the same representations across 0.50 R^2; when the readout chooses a push speed for a goal distance, V-JEPA 2 misses by 4 mm and the random ViT by 20 mm, no better than ignoring the object. Linear probes track none of this: a change in frame aggregation moves a probe more than pretraining does, and erasing the probed mass direction from the same representation costs nothing in one scene and 0.35 R^2 in the other. Whether a benchmark can rank models is an empirical property, and we give three checks that establish it.

物理推理视觉表征模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。