让视觉语言模型更快更稳，通过早期退出减少计算量。
FREE: Fast and Robust Vision Language Models with Early Exits
- 用对抗训练让中间层特征逼近最终层，支持快速退出。
- 实测推理速度提升1.51倍以上，准确率几乎不变。
- 适合对响应速度敏感的应用，如移动端或多轮交互。
近年来，视觉语言模型（VLMs）在视觉-语言任务中表现出显著性能提升，但其庞大模型规模给实际应用中的推理延迟带来挑战。为解决此问题，我们提出在VLMs中采用早期退出（EE）策略。然而，在标注数据有限的情况下训练退出分类器极具挑战。为此，我们引入FREE——一种基于GAN框架的对抗训练方法。每个退出节点包含一个Transformer层和一个分类器：前者被对抗训练以生成与最终层相似的特征表示，后者作为判别器。该方法实现输入自适应推理，在几乎不损失性能的前提下显著提升推理速度。实验表明，该方法有效缓解了过度思考和中期危机现象，提升了模型鲁棒性。我们验证了该方法使推理速度提升超过1.51倍，同时保持相近的性能水平。源代码已开源于https://github.com/Div290/FREE。
原文摘要 · Abstract (English)
In recent years, Vision-Language Models (VLMs) have shown remarkable performance improvements in Vision-Language tasks. However, their large size poses challenges for real-world applications where inference latency is a concern. To tackle this issue, we propose employing Early Exit (EE) strategies in VLMs. However, training exit classifiers in VLMs is challenging, particularly with limited labeled training data. To address this, we introduce FREE, an adversarial training approach within a GAN-based framework. Here, each exit consists of a transformer layer and a classifier. The transformer layer is adversarially trained to produce feature representations similar to the final layer, while a feature classifier serves as the discriminator. Our method focuses on performing input-adaptive inference that increases inference speed with minimal drop in performance. Experimental results demonstrate the effectiveness of our approach in enhancing accuracy and model robustness by mitigating overthinking and the phenomenon of mid-crisis that we highlight. We experimentally validate that our method speeds up the inference process by more than 1.51x while retaining comparable performance. The source code is available at https://github.com/Div290/FREE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。