不修改预训练权重,从冻结模型中恢复否定语义信息。
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

- 在中间层峰值处截取特征,利用注意力机制提取否定信号。
- 在COCO和VOC数据集上准确率分别提升35.1%和17.8%。
- 仅增加520万参数,适用于多种视觉-语言模型。
对比视觉-语言模型如CLIP将语义相反的短语(如“一只狗”与“不是一只狗”)映射到几乎相同的嵌入空间,对否定不敏感。我们将其归因于表征坍塌现象:通过追踪文本编码器中组合差异与视觉对齐的变化,发现中间层构建了组合语法,但最终层随着视觉对齐上升而坍塌该结构,导致最终表示忽略语法。为在不改变预训练权重的情况下恢复被丢失的否定信号,我们提出PeakPatch——一种轻量级后处理校正系统,它在编码器的组合性峰值处拦截特征。嵌入校正网络(ECN)使用交叉注意力从峰值层提取否定特异性信号,锚定稳定基线,并预测一个偏差向量,将丢失的语法重新注入最终层嵌入空间。互补的评分校正网络(SCN)预测判别任务中的有界标量偏移。两个模块联合端到端训练,所有CLIP参数保持冻结,仅增加520万参数(骨干的3.5%),并保留标准余弦相似度接口。在NegBench上,PeakPatch在COCO MCQ上达到74.3%(比CLIP高35.1%,比最佳微调方法高17.8%),在VOC MCQ上达65.5%,且在完全分布外否定检索任务上超越所有微调基线,尽管仅训练3.5%的参数。修正后的嵌入还迁移至文生图生成(否定得分提升18.4),并在ViT-B/32、ViT-L/14和SigLIP骨干上泛化良好。
原文摘要 · Abstract (English)
Contrastive vision-language models such as CLIP map semantically opposite phrases (e.g., "a dog" vs. "not a dog") to nearly identical embeddings, rendering them insensitive to negation. We attribute this failure to a phenomenon we call Representational Collapse: by tracking compositional divergence and visual alignment across the CLIP text encoder, we show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises, producing a syntax-blind final representation. To recover the lost negation signal without altering pretrained weights, we propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak. An Embedding Correction Network (ECN) uses cross-attention to extract a negation-specific signal from the peak layer, anchored to a stable baseline, and predicts a deviation vector that re-injects the lost syntax into the final-layer embedding space. A complementary Score Correction Network (SCN) predicts bounded scalar score offsets for discriminative tasks. Both modules are trained jointly end-to-end while all CLIP parameters remain frozen, adding only 5.2M parameters (3.5% of the backbone) and preserving the standard cosine similarity interface. On NegBench, PeakPatch achieves 74.3% on COCO MCQ (+35.1 over CLIP, +17.8 over the best encoder fine-tuning method) and 65.5% on VOC MCQ, while outperforming all fine-tuning baselines on fully out-of-distribution negation retrieval despite training only 3.5% of the parameters. The corrected embeddings also transfer to text-to-image generation (+18.4 negation score) and generalize across ViT-B/32, ViT-L/14, and SigLIP backbones. Project URL: https://stevencylu.github.io/PeakPatch/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。