视觉语言模型在多示例学习中会因示例过多而突然失效,且可通过轻量干预修复。
In-Context Collapse in Vision-Language Models and How to Mitigate it?

- 发现模型在增加示例时出现精度骤降,源于视觉与语言信息融合失败。
- 仅在早期/中间层添加适配器可恢复学习能力,准确率从0.39提升至0.91。
- 提出新方法CircA,一次训练即可让模型抵抗未知任务的崩溃现象。
多示例上下文学习(ICL)使视觉语言模型(VLMs)无需权重更新即可通过图像-标签示例进行适应,普遍认为示例越多效果越好。我们发现相反情况:随着示例增多,部分VLMs出现“上下文坍塌”——在合成分类、自然图像分类和VQA基准上精度急剧下降,某些模型甚至低于随机水平,但输出仍保持合理结构。在0.5B至11B参数的开放模型及前沿模型Claude Sonnet 4.5中,该现象呈梯度分布。两个能力可分离:对示例累积的鲁棒性与在上下文中学习新规则的能力,其组合形成三种可复现状态。参数匹配的病变与拯救实验因果定位坍塌根源为视觉-语言融合路径;在连接器及早期/中期层添加适配器可恢复真实学习(16样本下准确率由0.39提升至0.91),而同等容量的晚期读出适配器无效。我们提出 extsc{CircA},核心为一次性集成疫苗:仅在单一合成任务上训练一次,即可将抗坍塌能力迁移至未见任务族(在CIFAR/Fashion上从随机水平升至0.71/0.60)。最佳集成层并非最优权重巩固层,晚期读出以更少参数实现更高准确率与更低遗忘。坍塌本质是视觉-语言接口的整合失败,可通过轻量级、可迁移干预纠正。
原文摘要 · Abstract (English)
Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied. We show the opposite: as demonstrations accumulate, a subset of VLMs undergo an \emph{in-context collapse}, a sharp, sometimes catastrophic accuracy drop spanning synthetic classification, natural-image classification, and VQA benchmarks, in some models falling below chance while outputs remain well-formed. Across an open VLM panel ($0.5$B--$11$B) and a frontier model (Claude Sonnet 4.5), the collapse is graded. Two capabilities turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context, their combinations yield three reproducible regimes. A parameter-matched lesion-and-rescue causally localizes the collapse to the vision-language integration pathway: an adapter on the connector and early/mid layers restores genuine learning (remap accuracy $0.39!\rightarrow!0.91$ at 16 shots), while an equal-capacity adapter on the late readout does not. We propose \textsc{CircA}, whose core is a one-time integration vaccine: trained once on one synthetic task, it transfers collapse-resistance to unseen task families (chance$\rightarrow$$0.71$/$0.60$ on CIFAR/Fashion). The layers best for in-context integration are not the layers best for weight-based consolidation, the late readout achieves higher accuracy and less forgetting at fewer parameters. The collapse is an integration failure at the vision--language interface, correctable by a lightweight, transferable intervention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。