通过统一压缩与提取机制,大幅减少视觉模型推理时的显存和计算量。
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference

- 先自适应压缩冗余图像像素,再动态筛选关键视觉特征。
- 仅用10%视觉令牌,保留93.8%原始性能,首字延迟提速3.1倍。
- 无需训练,适配主流多模态模型,适合部署在资源受限场景。
视觉语言模型(VLMs)具备出色的视觉推理能力,但其推理成本随视觉标记数量激增。现有视觉标记剪枝方法存在两大局限:一是在视觉编码器后操作,未能优化编码阶段的高延迟;二是在严格标记预算下,难以兼顾整体视觉上下文与细粒度细节,导致性能下降。为此,我们提出PACE(Pixel-Adaptive Condense and Extract),一种无需训练的推理框架,通过统一的压缩与提取范式加速视觉编码器与大语言模型(LLM)。在压缩阶段,自适应像素压缩器(APC)在编码前评估视觉信息密度,自适应降采样冗余输入,减少编码计算,同时保留全局上下文与关键视觉线索。在提取阶段,动态双注意力提取器(DDAE)融合编码器内部视觉信号与LLM语义信号,选择性保留任务关键特征。将PACE集成至Qwen2.5-VL-7B后,仅使用10%视觉令牌即可保持93.8%原模型性能,首字延迟(TTFT)提升3.1倍。代码已开源:https://github.com/jjL357/PACE。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) demonstrate exceptional visual reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual tokens. Existing visual token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the vision encoder and the Large Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-Attention Extractor (DDAE) selectively retains visual tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual tokens, yielding a 3.1x speedup in time to first token (TTFT). Our code is available at https://github.com/jjL357/PACE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。