arXiv:2603.06459cs.CVcs.AI2026-03

冻结的视觉模型能精准感知连续几何,远超文本路径表现。

Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement

  • 用线性探测器从冻结特征中提取关节角度,精度达6.1度。
  • 文本输出误差高达20.0度,存在3.3倍性能瓶颈。
  • 无需微调,轻量探测即可实现多任务几何感知。

视觉语言模型编码了连续几何信息,但其文本路径无法有效表达:一个6,000参数的线性探测器从冻结特征中提取手部关节角度,平均绝对误差(MAE)为6.1度,而最佳文本输出仅达到20.0度,差距达3.3倍。使用LoRA微调(r=16,2,000张图像)将误差缩小至6.5度,表明问题在于训练路径而非表征能力。不同训练目标对精度影响大于架构差异:五种编码器(自监督、对比学习、混合范式)在代表性相似度仅0.41(CKA)的情况下,准确率均收敛至统计等效水平(R²≈0.55,TOST检验δ=0.03),体现功能收敛而非表征收敛。自回归生成会损害几何保真度,但根源在生成过程而非语言对齐;Qwen2.5-VL的LLM层反而提升探测精度。层分析显示所有架构在中间层(18-22层)出现精度峰值,注意力头携带显著几何信号。这些发现表明,单一冻结主干可通过轻量探测器实现多任务几何传感,无需微调或文本生成。

原文摘要 · Abstract (English)

Vision-language models encode continuous geometry that their text pathway fails to express: a 6,000-parameter linear probe extracts hand joint angles at 6.1 degrees MAE from frozen features, while the best text output achieves only 20.0 degrees -- a 3.3x bottleneck. LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees, providing evidence for a pathway-training deficit rather than a representational one. Training objective determines accuracy more than architecture: five encoders spanning self-supervised, contrastive, and hybrid paradigms converge to statistically equivalent accuracy (R^2 approximately 0.55, TOST-equivalent at delta=0.03) despite sharing as little as CKA=0.41 representational similarity -- functional convergence without representational convergence. Autoregressive generation damages geometric fidelity, but the damage originates in the generation process, not in language alignment: Qwen2.5-VL's LLM layers actually improve probe accuracy over its raw vision encoder. Layer-wise analysis reveals a universal mid-network accuracy peak across all architectures, with attention heads in layers 18-22 carrying disproportionate geometric signal. These findings enable a single frozen backbone to function as a multi-task geometric sensor through lightweight probes, without fine-tuning or text generation.

几何感知视觉语言模型冻结特征轻量探测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。