发现视觉语言模型中部分层会干扰任务性能，可通过动态跳过这些层提升效果。
Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models
- 通过干预单层参数，发现某些层反而降低下游任务表现。
- 提出任务-层交互向量，揭示不同任务对层干预的敏感性差异。
- 设计无需训练的测试时方法TaLo，可实时跳过干扰层，提升推理性能。
当前视觉语言模型（VLM）在多模态任务中表现出色，通常默认使用所有层进行预测。我们发现，对单一层进行干预（如置零参数）可在某些任务上提升性能，表明部分层会阻碍下游任务。通过系统性地对各层进行干预并测量性能变化，观察到绕过特定层可带来性能提升，且该现象在不同模型与数据集间具有泛化性，说明存在损害任务性能的‘任务干扰层’。为此，我们引入任务-层交互向量，量化每层对特定任务的影响。这些干扰层展现出任务相关的敏感性模式：需要相似能力的任务在层干预下响应趋势一致，其交互向量高度相似。基于此，我们提出无需训练、测试时使用的TaLo（任务自适应层跳过）方法，可动态识别并跳过最干扰的层。该方法无需参数更新，在多个模型和数据集上均有效，例如将Qwen-VL在ScienceQA地图任务上的准确率最高提升16.6%。本工作揭示了预训练VLM中未被注意的模块化特性，并提供了一种即插即用、免训练的推理阶段能力释放机制。源代码将公开。
原文摘要 · Abstract (English)
Current VLMs have demonstrated capabilities across a wide range of multimodal tasks. Typically, in a pretrained VLM, all layers are engaged by default to make predictions on downstream tasks. We find that intervening on a single layer, such as by zeroing its parameters, can improve the performance on certain tasks, indicating that some layers hinder rather than help downstream tasks. We systematically investigate how individual layers influence different tasks via layer intervention. Specifically, we measure the change in performance relative to the base model after intervening on each layer and observe improvements when bypassing specific layers. This improvement can be generalizable across models and datasets, indicating the presence of Task-Interfering Layers that harm downstream tasks' performance. We introduce Task-Layer Interaction Vector, which quantifies the effect of intervening on each layer of a VLM given a task. These task-interfering layers exhibit task-specific sensitivity patterns: tasks requiring similar capabilities show consistent response trends under layer interventions, as evidenced by the high similarity in their task-layer interaction vectors. Inspired by these findings, we propose TaLo (Task-Adaptive Layer Knockout), a training-free, test-time adaptation method that dynamically identifies and bypasses the most interfering layer for a given task. Without parameter updates, TaLo improves performance across various models and datasets, including boosting Qwen-VL's accuracy on the Maps task in ScienceQA by up to 16.6%. Our work reveals an unexpected form of modularity in pretrained VLMs and provides a plug-and-play, training-free mechanism to unlock hidden capabilities at inference time. The source code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。