发现视觉语言模型的辅助信号通道几乎总被关闭,提出用几何正则化保持关系推理能力。
The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models

- 通过几何损失在训练时正则化LoRA,仅前向传播保留辅助信号路径
- 保留几何路径可使关系类问题准确率维持原水平,不降反升
- 适用于需要稳定关系推理的下游任务,如视觉问答和空间推理
视觉语言模型中常引入可学习门控的辅助信号路径,但实验发现优化器几乎总是将其设为零:在五种注入设计中,所有门控路径在推理时均表现为行为关闭,即使门控参数理论上允许30%-45%信号通过,移除路径也对准确率无影响。我们归因于两种抑制机制:图像衍生信号对描述不变的梯度死亡现象,以及辅助信号反而损害损失函数的负效用。与其对抗抑制,不如加以利用:采用双曲视觉关系图(基于IoA驱动的蕴含锥与洛伦兹流形上的角度排斥)的几何辅助损失,仅在训练前向传播中使用,推理时丢弃。分解GQA按问题类型显示清晰分离:三种无几何损失配置在关系题上损失2.85-3.39个百分点,属性题提升约1个百分点;第四种虽有训练损失但推理用软提示,关系题仍下降5.14,属性题仅+0.23,表明仅训练正则化不足以保护关系性能。而保留几何路径的配置,关系准确率维持原水平并实现属性提升。在分布外数据集VSR上,RMS-prefix方案保持空间信号,去除几何损失(G2)导致性能下降4.6个百分点,确认其为OoD关键因素。次要发现:嵌入归一化对生成安全前缀注入至关重要,可学习门控应替换为固定非选通注入以匹配尺度。
原文摘要 · Abstract (English)
Auxiliary signal pathways in VLMs are routinely fitted with learnable gates so the optimiser can decide how much of the signal to admit. We find that the optimiser almost always decides on zero: across five injection designs, every gated pathway becomes behaviourally closed, with accuracy invariant to ablating the pathway at inference even when the gate parameter would nominally pass 30-45% of the signal. We attribute this suppression phenomenon to two regimes, a dead-gradient regime formalised through the caption-invariance of image-derived signals, and a negative-utility regime in which the auxiliary signal actively hurts the loss. Rather than fight suppression, we exploit it: we regularise LoRA fine-tuning with geometric auxiliary losses from hyperbolic visual relational graphs (IoA-driven entailment cones and angular repulsion on the Lorentz manifold), coupled only through the forward pass at training time and dropped at inference. Disaggregating GQA by question type exposes a clean dissociation. Three configurations without geometric losses at inference lose 2.85-3.39pp on relational questions while gaining ~1pp on attribute questions; a fourth that trains with the losses but infers through a soft prompt loses 5.14pp on rel for only +0.23pp on attr, so training-time regularisation alone does not protect relational accuracy without a geometric inference pathway. Configurations that keep the geometric pathway at inference preserve vanilla-level relational accuracy and match the attribute gain. Out of distribution on VSR, the RMS-prefix recipe preserves the spatial signal; stripping the geometric losses (G2) collapses VSR by 4.6pp, isolating them as the OOD source. A secondary result: embedding-norm alignment is necessary for generation-safe prefix injection, and learnable gates should be replaced with fixed, non-optional injection at matched scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。