提出一种无需训练的视觉令牌压缩方法，显著提升多模态大模型效率。
PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models
- 基于提示感知策略动态识别关键视觉令牌并聚类。
- 减少83%计算量，压缩率89%，准确率保留97%。
- 适合追求高效推理的多模态模型部署场景。
多模态大语言模型在视觉任务中表现强劲，但处理长上下文时面临巨大计算与内存压力。为此，本文提出PAR（Prompt-Aware Token Reduction），一种无需额外训练、可即插即用的视觉令牌压缩方法。不同于依赖注意力机制且忽略跨模态交互的现有方法，PAR采用提示感知策略，自适应识别并聚类关键视觉令牌。将视觉冗余分为外部与内部两类：外部冗余通过语义检索消除，内部冗余则由令牌路由机制处理。实验表明，在多个视觉问答任务中，PAR实现83%的浮点运算量（FLOPs）降低，压缩率达89%，同时保持97%的基线准确率。其自适应设计使令牌压缩比达2倍，显著优于先前方法，实现了性能与效率的更好平衡。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) demonstrate strong performance across visual tasks, but their efficiency is hindered by significant computational and memory demands from processing long contexts in multimodal inputs. To address this, we introduce PAR (Prompt-Aware Token Reduction), a novel and plug-and-play approach that reduces visual tokens efficiently without compromising model performance. Unlike previous methods that rely heavily on attention mechanisms and overlooking cross-modal interactions , we uses a prompt-aware strategy to adpative identify and cluster essential visual tokens. PAR categorizes visual context redundancy into two types: external and internal. External redundancy is minimized through semantic retrieval, while internal redundancy is addressed using a token routing mechanism. This method substantially reduces computational load without requiring additional training or complex architectural modifications. \textbf{Experimental results demonstrate that across various visual question answering tasks, PAR reduces FLOPs by 83\% with a compression ratio of 89\%, while retaining 97\% of baseline accuracy.} The adaptive design of PAR achieves a 2x token reduction ratio compared to prior approaches, enabling a better balance between performance and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。