通过文本驱动的视觉标记剪枝，显著降低大模型推理开销。
CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference
- 先于大模型处理，根据文本指令筛选相关视觉标记
- 在极端剪枝下仍保持99.5%准确率，推理延迟减半以上
- 适合资源受限场景下的高效视觉语言模型应用
大型视觉语言模型（LVLM）通常需处理数百至数千个视觉标记，导致显著的推理开销。现有剪枝方法或在大模型前使用与文本无关的启发式策略，或在大模型内部剪枝，代价是效率下降和跨模态注意力噪声增加。为此，我们提出CRISP，一种预大模型但由文本驱动的视觉标记剪枝框架，兼顾指令相关证据与必要场景上下文。该框架采用两阶段流程：第一阶段识别与文本对齐的视觉标记；第二阶段通过语义多样性增强上下文完整性。在LLaVA-1.5和LLaVA-NeXT上的大量实验表明，CRISP在激进剪枝比例下仍能保持优异性能，实现高达99.5%的准确率，同时将推理成本和延迟降低超过2倍。CRISP为高效LVLM推理提供了实用解决方案，尤其适用于资源受限场景。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) typically require processing hundreds to thousands of visual tokens, leading to substantial inference overhead. Existing visual token pruning methods either operate before the LLM using text-agnostic heuristics or prune inside the LLM at the cost of efficiency and noisy cross-modal attention. To address these limitations, we propose CRISP, a pre-LLM yet text-driven visual token pruning framework that preserves both instruction-relevant evidence and essential scene context. CRISP works in a two-stage pipeline: Stage 1 first identifies text-aligned visual tokens, and Stage 2 enhances contextual completeness through semantic diversity. Extensive experiments on LLaVA-1.5 and LLaVA-NeXT demonstrate that CRISP achieves superior performance retention under aggressive pruning ratios, maintaining up to 99.5% accuracy while reducing inference cost and latency by more than 2 times. CRISP serves as a practical solution for efficient LVLM inference, especially in resource-constrained scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。