轻量级视觉语言模型推理流水线，实现实时低延迟部署。
LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments
- 通过图像块筛选、令牌压缩与推测解码联合优化
- 在自动驾驶平台实现2.5倍端到端延迟降低
- 适合机器人、车载系统等资源受限场景
本文提出一种专为嵌入式设备（如机器人和自动驾驶）优化的高效视觉语言模型（VLM）推理流水线。该流水线通过联合使用图像块选择以过滤无关摄像头视图、令牌选择模块以缩短LLM输入序列长度，以及推测解码加速令牌生成，显著降低计算开销。在NVIDIA DRIVE Thor平台上的评估显示，该方案在不损失任务准确率的前提下，实现了2.5倍的端到端延迟降低；当结合FP8后训练量化时，速度提升进一步达到3.2倍。结果表明，该流水线是资源受限环境下实现VLM实时部署的可行方案。
原文摘要 · Abstract (English)
This paper introduces an efficient Vision-Language Model (VLM) pipeline specifically optimized for deployment on embedded devices, such as those used in robotics and autonomous driving. The pipeline significantly reduces the computational overhead by jointly leveraging patch selection to filter irrelevant camera views, a token selection module to reduce input sequence length for the LLM, and speculative decoding to accelerate token generation. Evaluation on the NVIDIA DRIVE Thor platform for automonous driving application, our pipeline achieves $2.5\times$ end-to-end latency reduction without compromising task accuracy. The speed-up further increases to $3.2\times$ when applying FP8 post-training quantization. These results demonstrate our pipeline as a viable solution for enabling real-time VLM deployment in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。