系统梳理视觉语言模型推理加速技术，助你高效部署大模型。
Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
- 按视觉令牌压缩、内存管理、架构设计、解码策略四维度分类优化方法
- 揭示高分辨率输入下注意力机制二次复杂度的性能瓶颈
- 适合关注多模态模型落地与推理效率的研究者和工程师
尽管大型视觉语言模型（LVLMs）展现出强大的多模态推理能力，但其可扩展性与部署受限于巨大的计算需求。特别是高分辨率输入数据带来的海量视觉标记，加剧了因注意力机制具有二次复杂度而引发的性能问题。为应对这一挑战，研究界已提出多种优化框架。本文全面综述当前最先进的LVLM推理加速技术，提出一个系统的分类体系，将现有优化框架归纳为四个主要维度：视觉标记压缩、内存管理与服务、高效架构设计以及先进解码策略。此外，我们批判性分析这些方法的局限性，并识别关键开放问题，以启发未来高效多模态系统的研究方向。
原文摘要 · Abstract (English)
Although Large Vision Language Models (LVLMs) have demonstrated impressive multimodal reasoning capabilities, their scalability and deployment are constrained by massive computational requirements. In particular, the massive amount of visual tokens from high-resolution input data aggravates the situation due to the quadratic complexity of attention mechanisms. To address these issues, the research community has developed several optimization frameworks. This paper presents a comprehensive survey of the current state-of-the-art techniques for accelerating LVLM inference. We introduce a systematic taxonomy that categorizes existing optimization frameworks into four primary dimensions: visual token compression, memory management and serving, efficient architectural design, and advanced decoding strategies. Furthermore, we critically examine the limitations of these current methodologies and identify critical open problems to inspire future research directions in efficient multimodal systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。