破解视觉语言模型推理效率瓶颈，系统梳理优化路径。
Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
- 按编码、预填充、解码流程分析效率瓶颈
- 揭示高分辨率特征与注意力开销的协同制约
- 适合关注模型部署与系统优化的研究者
大型视觉语言模型（LVLMs）能对图像和视频进行复杂推理，但其推理受制于称为视觉标记主导的系统性效率障碍。该问题源于高分辨率特征提取、二次方注意力扩展与内存带宽限制之间的多阶段相互作用。本文提出一个围绕推理生命周期的效率技术分类体系，涵盖编码、预填充和解码三个阶段。不同于以往针对单一优化的综述，本研究分析端到端流程，揭示上游决策如何决定下游瓶颈：计算密集型视觉编码、大规模上下文预填充的高负载，以及带宽受限解码中的‘视觉内存墙’。通过将效率图景分解为信息密度调控、长上下文注意力管理、内存限制突破三个维度，本文系统阐释了孤立优化如何协同应对视觉保真度与系统效率间的权衡。最后基于初步实证洞察，提出四个未来方向：基于功能单元敏感性的混合压缩、放宽验证的模态感知解码、面向流式连续性的渐进状态管理，以及通过软硬件协同设计实现的阶段解耦服务。文献库地址：https://github.com/SuDIS-ZJU/Efficient-LVLMs-Inference。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance. This overhead is driven by a multi-regime interplay between high-resolution feature extraction, quadratic attention scaling, and memory bandwidth constraints. We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding, prefilling, and decoding. Unlike prior reviews focused on isolated optimizations, we analyze the end-to-end pipeline to reveal how upstream decisions dictate downstream bottlenecks, covering compute-bound visual encoding, the intensive prefilling of massive contexts, and the ''visual memory wall'' in bandwidth-bound decoding. By decoupling the efficiency landscape into the axes of shaping information density, managing long-context attention, and overcoming memory limits, this work provides a structured analysis of how isolated optimizations compose to navigate the trade-off between visual fidelity and system efficiency. The survey concludes by outlining four future frontiers supported by pilot empirical insights, including hybrid compression based on functional unit sensitivity, modality-aware decoding with relaxed verification, progressive state management for streaming continuity, and stage-disaggregated serving through hardware-algorithm co-design. Our literature repository is at https://github.com/SuDIS-ZJU/Efficient-LVLMs-Inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。