用轻量草案模型加速视觉语言模型推理，速度提升1.55至1.85倍。
FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
- 用轻量草案模型预生成词元，全模型非自回归验证并指导修正。
- 相比原模型最后一层，推理速度提升1.55至1.85倍，性能损失极小。
- 适合需要低延迟视觉问答或图文生成的实时应用。
视觉语言模型（VLMs）在视觉理解与问答生成方面取得显著进展，但常因自回归解码导致计算成本高、推理延迟大。本文提出基于模仿学习的自推测解码（SSD）框架FastVLM，以缓解此问题。该方法采用轻量级草案模型进行自回归词元生成，同时由完整模型非自回归地验证这些词元。被接受的词元直接通过，被拒绝的则由完整模型修正，并用于引导草案模型优化。通过模仿网络，FastVLM将完整模型的深层结构洞察融入草案模型训练中，实现效率与准确性的平衡。实验表明，该方法相较原模型最终层推理速度提升1.55至1.85倍，性能损失可忽略。
原文摘要 · Abstract (English)
Vision-language Models (VLMs) have made significant strides in visual understanding and query response generation, but often face challenges of high computational cost and inference latency due to autoregressive decoding. In this work, we introduce an imitation-learning-based Self-Speculative Decoding (SSD) framework, named FastVLM, to address these limitations. Our approach employs a lightweight draft model for token generation in an autoregressive manner, while a full model verifies these tokens non-autoregressively. Accepted tokens proceed seamlessly, while rejected tokens are corrected by the full model and used to guide the draft model's refinement. Through an imitation network, FastVLM enhances the draft model by integrating deeper level insights from the full model's architecture. Also, it maintains the performance integrity of the full model while training the draft model, achieving a balance between efficiency and accuracy. Our method speeds up the inference process by 1.55-1.85x as compared to the final layer with minimal loss in performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。