arXiv:2608.07193cs.LGcs.CV2026-08

用AI自动生成视觉标记剪枝策略,大幅降低大模型推理开销。

An AI4AI Framework for Visual Token Pruning

论文配图:An AI4AI Framework for Visual Token Pruning
图 1 · 摘自论文原文
  • 基于专用语言构建搜索空间,引导LLM高效设计剪枝策略。
  • 剪掉94.4%视觉标记,仍保持99%性能,推理耗时降6.4倍。
  • 无需训练,适用于多种模型和任务,通用性强。

视觉标记剪枝可显著降低多模态大语言模型(MLLM)的推理成本,但现有方法多依赖固定的手工启发式规则和昂贵的专家试错。随着剪枝目标、预算和模型架构多样化,手动探索设计空间日益困难。本文提出一种AI4AI框架,回答核心问题:大语言模型能否自动设计有效的视觉标记压缩算法?尽管大模型具备广泛算法知识和推理能力,但将通用知识转化为特定任务的有效解仍具挑战。关键在于设计合适的搜索状态表示,连接模型内部知识与剪枝的结构要求。为此,我们提出AutoPrune——一种无需训练的LLM驱动剪枝策略设计框架。其核心是131个可复用原子构成的视觉标记剪枝领域专用语言(TPDSL),涵盖预算控制、标记评分、选择约束与标记重组。TPDSL将每个搜索状态表示为强基线策略的残差修改,缩小搜索空间并聚焦于影响性能的关键组件。在14个多模态基准和3种MLLM骨干网络上的实验表明,AutoPrune在去除94.4%视觉标记的情况下,性能损失低于1%,同时减少9.9倍计算量(FLOPs)和6.4倍预填充延迟。

原文摘要 · Abstract (English)

Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error. As pruning objectives, budgets, and model architectures diversify, manually navigating the expanding design space becomes increasingly difficult. This paper aims to build an AI4AI framework for visual-token pruning by addressing a natural question: Can large language models automatically design effective visual-token reduction algorithms? Although LLMs possess broad algorithmic knowledge and strong reasoning capabilities, translating such general knowledge into effective solutions for a specialized task remains nontrivial. We argue that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-token pruning. Based on this insight, we propose AutoPrune, a training-free framework for LLM-driven visual-token pruning policy design. At its core, AutoPrune introduces a Token Pruning Domain-Specific Language (TPDSL) comprising 131 reusable atoms for budget control, token scoring, selection constraints, and token reassembly. A key property of TPDSL is that it represents each search state as a residual modification of a strong base policy. This residual formulation narrows the search space and directs the LLM's attention toward the policy components that are most consequential for performance. Experiments on 14 multimodal benchmarks and three MLLM backbones demonstrate the effectiveness, efficiency, and transferability of AutoPrune. Even when removing 94.4% of visual tokens, AutoPrune preserves more than 99% of full-token performance while reducing FLOPs by 9.9x and prefill latency by 6.4x.

视觉剪枝AI4AI大模型优化提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。