通过预测编码剪枝冗余图像块，显著加速文档与界面理解的视觉模型推理。
PixelPrune: Pixel-Level Adaptive Visual Token Reduction via Predictive Coding
- 在像素级预压缩图像，用预测编码识别并移除重复块。
- 在多个基准上保持精度的同时，推理速度提升4.2倍，训练加速1.9倍。
- 无需训练、无参数，支持无损与可控有损压缩，适合部署场景。
文档理解与GUI交互是视觉语言模型（VLMs）最具价值的应用之一，但其计算开销极大：细粒度文本和小尺寸UI元素需要高分辨率输入，生成数以万计的视觉标记。我们观察到这种开销大多浪费——在文档与GUI基准中，图像块的像素唯一性仅为22%–71%，其余为同一图像内完全重复的块。本文提出PixelPrune，基于预测编码实现像素级冗余压缩，在视觉变换器（ViT）编码器前剪枝冗余块。因其在神经计算前于像素空间操作，可同时加速ViT编码器与下游大语言模型，覆盖完整推理流程。该方法无需训练、无学习参数，支持像素无损压缩（τ=0）及可控有损压缩（τ>0）。在三种模型规模及文档与GUI基准上的实验表明，PixelPrune在保持竞争力任务准确率的同时，实现最高4.2倍推理加速与1.9倍训练加速。代码已开源：https://github.com/OPPO-Mente-Lab/PixelPrune。
原文摘要 · Abstract (English)
Document understanding and GUI interaction are among the highest-value applications of Vision-Language Models (VLMs), yet they impose exceptionally heavy computational burden: fine-grained text and small UI elements demand high-resolution inputs that produce tens of thousands of visual tokens. We observe that this cost is largely wasteful -- across document and GUI benchmarks, only 22--71\% of image patches are pixel-unique, the rest being exact duplicates of another patch in the same image. We propose \textbf{PixelPrune}, which exploits this pixel-level redundancy through predictive-coding-based compression, pruning redundant patches \emph{before} the Vision Transformer (ViT) encoder. Because it operates in pixel space prior to any neural computation, PixelPrune accelerates both the ViT encoder and the downstream LLM, covering the full inference pipeline. The method is training-free, requires no learnable parameters, and supports pixel-lossless compression ($τ{=}0$) as well as controlled lossy compression ($τ{>}0$). Experiments across three model scales and document and GUI benchmarks show that PixelPrune maintains competitive task accuracy while delivering up to 4.2$\times$ inference speedup and 1.9$\times$ training acceleration. Code is available at https://github.com/OPPO-Mente-Lab/PixelPrune.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。