视觉语言模型的辅助损失实际影响的是语言模型，而非视觉隐变量。
Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
- 用线性探测和扰动测试发现，显式隐变量可被绕过
- 隐变量与视觉目标的余弦相似度越强，准确率反而越低（r=-0.94）
- 适合关注模型内部机制与训练目标真实影响的研究者
潜空间视觉推理（LVR）在视觉语言模型（VLMs）中于感知与答案生成间插入受监督的隐变量。当前方法普遍以这些隐变量与其视觉目标之间的余弦相似度或均方误差（MSE）作为训练损失与质量指标，假设对齐程度越高，答案越准。我们设计了五种LVR变体进行测试，发现该假设完全反转：所有五种情况下，余弦对齐度与准确率呈负相关（r = -0.94）。为解释此现象，我们提出PRISM，一对推理时诊断工具：线性探测用于判断答案何时可解码，扰动测试用于判断隐变量是否承重。结果显示，受监督隐变量大多被绕过；扰动它们最多使准确率下降4个百分点。答案可在隐变量下游解码，但无法在其处解码，该解码差距大小可预测各变体对隐变量的依赖程度。这与信息瓶颈视角一致：辅助目标通过共享参数重塑语言模型，而非优化其名义上的隐变量。
原文摘要 · Abstract (English)
Latent visual reasoning (LVR) inserts supervised latent tokens between perception and answer generation in vision-language models (VLMs). The field uses alignment between these latents and their visual targets, i.e., cosine similarity or mean squared error (MSE), as both the training loss and the quality metric, assuming that better alignment yields a better answer. We test this with a designed matrix of five LVR variants and find the assumption inverted: cosine alignment is negatively correlated with accuracy across all five (r=-0.94). To explain this, we introduce PRISM, a pair of inference-time diagnostics: a linear probe that asks where the answer is decodable, and a corruption test that asks whether the latent is load-bearing. The supervised latents are largely bypassed. Corrupting them shifts accuracy by at most four points. The answer is decodable downstream of the latent but not at it, and the size of this decodability gap predicts how much each variant relies on its latent under perturbation. Consistent with an Information Bottleneck reading of the loss, the auxiliary objective reshapes the language model via shared parameters rather than via the latent variable it nominally optimizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。