arXiv:2607.21973cs.CV2026-07中稿 · ACM MM 2026被引 4

提出PIB框架,让视觉大模型适配更高效精准。

Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation

论文配图:Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation
图 1 · 摘自论文原文
  • 基于信息瓶颈原理,分层调控任务相关与无关信息
  • 34个数据集上平均仅调0.35%参数,最高达93.01%准确率
  • 适合需要轻量适配和强鲁棒性的视觉任务应用

视觉基础模型常作为冻结主干用于下游视觉识别,参数高效适配成为核心问题。提示调优(如VPT)虽轻量,但其分层行为仍不明确:性能对提示深度、位置和任务分布敏感,且标准基准上的提升未必带来稳健泛化。本文认为该局限不仅是优化问题,更是分层信息分配问题——现有方法缺乏对提示引导表征中应保留、抑制和传递内容的系统控制。受信息瓶颈原理启发,我们提出提示信息瓶颈(PIB),通过正则化分层压缩-充分性权衡,构建更连贯的跨层信息路径。有效适配应最小但足够,早期层保留任务相关局部证据,深层逐步丢弃干扰因素与冗余细节。大量实验表明,PIB在34个数据集上表现优异,达到FGVC 92.1%、HTA 93.01%、VTAB-1k 77.33%准确率,平均仅调整0.35%参数。除基准精度外,PIB还解释了提示容量非单调增长现象,减少捷径依赖,并提升分布偏移与细粒度识别下的鲁棒性。结果表明PIB既是实用方法,也是冻结视觉模型适配的信息分配新视角。代码已开源。

原文摘要 · Abstract (English)

Vision foundation models are increasingly reused as frozen backbones for downstream visual recognition, making parameter-efficient adaptation a central problem. Prompt-based adaptation, including Visual Prompt Tuning (VPT), provides a lightweight way to specialize these models, but its layer-wise behavior remains poorly understood: performance is sensitive to prompt depth, placement, and task distribution, and gains on standard in-domain benchmarks do not always translate into robust generalization. We argue that this limitation is not solely an optimization issue, but a layer-wise information allocation issue: existing prompt-based methods lack principled control over what prompt-conditioned representations should preserve, suppress, and propagate across depth. Inspired by the Information Bottleneck principle, we introduce Prompted Information Bottlenecks (PIB), a framework that regularizes layer-wise compression-sufficiency trade-offs and promotes a more coherent cross-layer information path. The key idea is that effective adaptation should be minimal yet sufficient, retaining task-relevant local evidence in earlier layers while progressively discarding nuisance factors and redundant details in deeper layers. Extensive experiments show that PIB achieves strong performance across 34 datasets, reaching 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k, while tuning only 0.35% parameters on average across the main settings. Beyond benchmark accuracy, PIB helps explain the non-monotonic behavior of prompt capacity scaling, reduces shortcut reliance, and improves robustness under distribution shift and fine-grained recognition settings. These results position PIB as both a practical method and an information-allocation perspective for adapting frozen vision foundation models. Our code is available at https://github.com/itsnotacie/MM-26-PIB

视觉模型提示调优信息瓶颈轻量适配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。