通过挖掘模型浅层激活值，实现零训练快速分类。
Taking Shortcuts for Categorical VQA Using Super Neurons
- 直接探测大语言模型浅层的标量激活值，找到高效判别神经元。
- 在首个生成标记处即可完成分类，速度提升最高达5.10倍。
- 无需微调，适合追求推理加速与轻量部署的场景。
稀疏注意力向量（SAVs）已成为改进视觉语言模型（VLMs）性能的无训练替代方案，无需监督微调或低秩适配。本文受此启发，发现直接探测VLM原始激活的标量值，即可在多样化的视觉基础下游任务中构建准确分类器。与关注注意力向量不同，转向标量激活显著扩大了可搜索的参数空间，使我们能在首个生成标记处立即找到更具判别力的神经元，称之为超神经元（Super Neurons, SNs）。在探针设置下，我们发现足够数量的SNs存在于大型语言模型的浅层，从而支持在第一个模型层即实现极端早期退出。相较于原模型，SNs在保持鲁棒性的同时，分类性能显著提升，并实现高达5.10倍的速度加速。
原文摘要 · Abstract (English)
Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of Vision Language Models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model's prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。