用热流方法保守压缩视觉令牌，不丢信息还保结构
Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models
- 通过谱热流计算稳定重要性，保持空间结构一致
- 88.9%令牌被压缩后，模型仍保留95.6%原始性能
- 无需微调，兼容FlashAttention，适合高效推理场景
视觉语言模型（VLMs）推理成本高，因需处理长序列视觉令牌。现有令牌剪枝方法在高压缩率下常因盲目丢弃信息而失效，破坏空间结构或导致多样性丧失。我们提出SpecFlow，一种无需训练的框架，将剪枝范式从破坏性删除转向保守凝聚，严格保证空间覆盖与统计守恒以确保稳定性。将视觉令牌视为kNN图中的节点，SpecFlow (i) 通过谱热流计算稳定的重要性场以保持结构连贯性，(ii) 采用自适应空间划分分配预算以保障覆盖，(iii) 将被舍弃的信息聚合至核心集汇点以维持统计守恒。该方法即插即用，无需微调，兼容FlashAttention。实验表明，SpecFlow在多种任务、模型架构和剪枝比例下均优于当前最优方法。值得注意的是，经SpecFlow压缩后，LLaVA-1.5在剔除88.9%视觉令牌的情况下仍保持95.6%原始性能，展现出卓越的效率-精度平衡。代码已公开于https://github.com/Lzy-dot/SpecFlow。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are costly at inference time because they must process long sequences of visual tokens. Existing token pruning methods often degrade under high compression by blindly discarding information, breaking spatial structure or collapsing diversity. We propose SpecFlow, a training-free framework that shifts the paradigm from destructive pruning to conservative condensation, strictly enforcing spatial coverage and statistical conservation to ensure stability. Treating visual tokens as nodes in a $k$NN graph, SpecFlow (i) computes a stable importance field via spectral heat flow to preserve structural coherence, (ii) allocates budgets via adaptive spatial partitioning to guarantee coverage, and (iii) aggregates discarded information into coreset sinks to maintain statistical conservation. The method is plug-and-play, requires no fine-tuning, and is compatible with FlashAttention. Experiments confirm that our SpecFlow outperforms SOTA methods across tasks, VLM architectures, and pruning ratios. Notably, LLaVA-1.5 with SpecFlow retains 95.6% of original performance despite pruning 88.9% of visual tokens, offering an exceptional efficiency-accuracy balance. Code is available at https://github.com/Lzy-dot/SpecFlow
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。