提出F³A方法，在不训练模型前提下高效分配视觉标记预算。
How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
- 用冻结的稀疏感知头匹配问题条件与图像块，实现动态筛选。
- 在固定视觉标记预算下，通过局部精炼和覆盖竞争提升关键信息保留率。
- 无需训练或额外推理，适合大规模多模态模型压缩部署。
视觉语言模型通过向语言主干输入越来越长的视觉标记序列来提升感知能力，但随之而来的推理开销引发了一个基础性扩展问题：随着多模态模型规模增大，究竟需要多少视觉标记？在固定视觉标记预算下又该如何分配？现有无训练剪枝方法通常依赖单次代理策略，如解码器注意力、视觉相似性或条件多样性。我们主张将视觉标记剪枝视为任务相关的证据搜索，尤其在激进压缩和跨模型尺度下更适用。本文提出F³A——一种在语言模型处理图像标记前运行的无训练路由机制。F³A构建轻量级问题条件线索，通过冻结的稀疏感知头将其匹配到视觉网格标记，并利用粗粒度证据定位、局部精炼、覆盖保持的竞争机制以及未覆盖区域的恢复策略，在固定视觉标记预算下进行高效分配。该方法无需模型训练，无需额外大语言模型前向传播，且完全兼容原始多模态提示与解码流程。
原文摘要 · Abstract (English)
Vision-language models improve perception by feeding increasingly long visual token sequences into language backbones, but the resulting inference cost raises a basic scaling question: as multimodal models grow, how many visual tokens are actually needed, and how should they be allocated under a fixed visual token budget? Existing training-free pruning methods typically answer this with one-shot proxies such as decoder attention, visual similarity, or conditional diversity. We argue that visual token pruning is better viewed as task-conditioned evidence search, especially under aggressive compression and across model scales. We propose F^3A, a training-free router for visual token pruning that operates before the language model consumes image tokens. F^3A builds lightweight question-conditioned cues, matches them to visual-grid tokens through frozen sparse sensing heads, and allocates a fixed vision token budget via coarse evidence localization, local refinement, coverage-preserving competition, and recovery of under-covered regions. It requires no model training, no extra LLM forward pass and preserves the original multimodal prompting and decoding pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。