用大模型自身压缩视觉令牌，高效降耗还提性能。
VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression
- 让预训练大模型当压缩器，不加新模块也不重训。
- 高压缩比下表现更优，单令牌时仍稳定有效。
- 压缩后还能提升原模型，适合追求效率的视觉任务。
视觉语言模型（VLMs）处理大量视觉令牌，导致推理延迟和内存开销显著。现有压缩方法中，免训练策略依赖启发式度量，高压缩比下性能下降严重；基于训练的方法引入外部压缩模块，迫使VLM主干适应，需大量重训练且破坏预训练先验。有效的视觉令牌压缩依赖强信息编码能力，而这一能力在预训练VLM中已存在，却被现有方法忽视。为此，我们提出VisCo，一种训练高效的自压缩框架，将预训练VLM自身作为内在压缩器。VisCo是一个参数共享的自编码器，利用少量记忆令牌压缩视觉信息，并在编码到解码间传递层次化信息。实验表明，VisCo在所有评估压缩比下均超越先前方法，尤其在激进压缩下增益更大，在极端单令牌设置下仍保持稳定。此外，结合原始视觉令牌后，学习到的记忆令牌甚至能提升基础模型表现，说明VisCo捕获了超出压缩需求的互补表征。代码已开源。
原文摘要 · Abstract (English)
Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression. Code is available at: \href{https://github.com/Zyvpeng/VisCo}{\textcolor{blue}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。