提出新剪枝方法，在极低视觉令牌下仍保持模型性能稳定。
DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservation
- 通过分布一致性度量识别剪枝导致的特征偏移
- 在仅保留16个视觉令牌时，保持LLaVA-1.5-7B 92.1%的性能上限
- 适合资源受限场景下的高效多模态模型部署
近期视觉令牌剪枝方法在中等令牌预算下能有效保持模型性能，但在超低预算下变得不稳定。分析发现，随着剪枝预算降低，准确率下降常伴随特征分布显著偏移，且分布偏移程度与性能退化强相关。为此，我们引入轻量级分布一致性度量，用于估计保留令牌与完整令牌间的分布差异。基于此，提出两阶段剪枝框架：锚点-上下文图恢复（ACGR）在剪枝前传递上下文信息，文本感知令牌聚类选择（TATCS）在检测到严重分布偏移时动态重选代表性令牌。大量实验表明，该方法在超低令牌预算下表现更优且更稳定。值得注意的是，在仅保留16个视觉令牌时，其在LLaVA-1.5-7B上仍保持92.1%的上限平均性能。
原文摘要 · Abstract (English)
Recent vision token pruning methods effectively preserve model performance under moderate token budgets but become unstable under ultra-low token budget. Our analysis shows that as the pruning budget decreases, accuracy degradation is often accompanied by larger feature distribution shifts. Critically, the degree of this distribution shift strongly correlates with performance degradation. To better characterize this phenomenon, we introduce a lightweight distribution consistency metric to estimate the distribution shift between retained and full tokens. Motivated by these observations, we propose a two-stage pruning framework consisting of Anchor-Context Graph Recovery (ACGR) and Text-Aware Token Cluster Selection (TATCS). Specifically, ACGR transfers contextual information before token removal, while TATCS dynamically re-selects representative tokens when severe distribution shift is detected. Extensive experiments demonstrate that our method achieves superior and more stable performance under ultra-low token budget. Notably, it retains 92.1% of the upper-bound average performance on LLaVA-1.5-7B with only 16 visual tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。