通过微调注意力头缩放，用千级参数显著提升视觉语言模型的空间推理能力。
ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models
- 仅用1000个可训练参数，通过缩放特定层的注意力和MLP激活来改进模型
- 在SpatialEval上相对准确率提升最高达134.1%，接近标准方法性能
- 适用于需要轻量适配的场景，尤其适合资源受限环境下的模型优化
空间推理仍是众多视觉语言模型（VLMs）的持续挑战，传统改进方式通常需大量额外参数进行微调。初步分析发现，仅通过重缩放选定Transformer层的激活值（不修改预训练权重），即可显著影响下游性能。受此启发，我们提出ScAle：一种超轻量级适配方法，通过学习少量标量系数，对冻结主干网络中最后一令牌的注意力和MLP激活进行调制。我们在合成空间推理基准SpatialEval及真实世界VQA数据集COCOQA和VGQA上评估该方法，涵盖多个模型家族。结果表明，ScAle仅使用1,000个可训练参数，即可实现最高134.1%的相对准确率提升，远超标准参数高效微调方法（如LoRA）所需的数百万参数。尽管极简，其表现仍恢复了标准PEFT方法的大部分性能，同时保持了强非空间推理能力。这证明，有界激活重加权是一种简单、架构无关且高度参数高效的VLM适配替代方案。
原文摘要 · Abstract (English)
Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters. Our preliminary analysis reveals that rescaling activations in selected transformer layers-without modifying pretrained weights-can significantly influence downstream performance. Motivated by this observation, we propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coefficients to modulate last-token attention and MLP activations in a fully frozen backbone. We evaluate our method on the synthetic spatial reasoning benchmark SpatialEval and on real-world VQA datasets (COCOQA and VGQA) across multiple model families. Our method, ScAle, achieves up to 134.1% relative accuracy gains using only 1K trainable parameters without requiring millions of trainable parameters as in standard PEFT methods such as LoRA. Despite its extreme compactness, our approach recovers a substantial fraction of standard PEFT performance while preserving strong non-spatial VQA accuracy. These results demonstrate that bounded activation reweighting provides a simple, architecture-agnostic, and highly parameter-efficient alternative for adapting pretrained VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。