用视觉方法压缩大模型输入，让推理更快更省资源。
Vision-centric Token Compression in Large Language Model
- 将远距离文本转为图像，由轻量视觉模型快速过滤无关信息。
- 在11个任务上用2.3倍少的令牌达到相同准确率，内存减半。
- 适合需要长上下文处理但算力有限的场景，如文档理解。
现实应用正将上下文窗口扩展至数十万令牌，而大语言模型参数量也从数十亿增至万亿级，导致计算与内存成本飙升，使令牌压缩变得不可或缺。本文提出视觉中心令牌压缩（Vist），一种模拟人类阅读的快慢路径框架：快路径将远距离令牌渲染为图像，由冻结的轻量视觉编码器快速扫描低显著性内容；慢路径则将邻近窗口送入大模型进行细粒度推理。通过概率感知视觉增强（PVE）目标，在训练中屏蔽高频词，引导重采样器聚焦语义丰富区域——正如熟练读者跳过虚词。在11个上下文学习基准测试中，Vist以2.3倍更少的令牌实现同等准确率，减少16%浮点运算和50%内存占用。该方法优于最强文本编码器压缩方法CEPE，在TriviaQA、NQ、PopQA、NLUI、CLIN等基准上平均提升7.6%，为大模型令牌效率树立新标准。项目地址：https://github.com/CSU-JPG/VIST。
原文摘要 · Abstract (English)
Real-world applications are stretching context windows to hundreds of thousand of tokens while Large Language Models (LLMs) swell from billions to trillions of parameters. This dual expansion send compute and memory costs skyrocketing, making token compression indispensable. We introduce Vision Centric Token Compression (Vist), a slow-fast compression framework that mirrors human reading: the fast path renders distant tokens into images, letting a frozen, lightweight vision encoder skim the low-salience context; the slow path feeds the proximal window into the LLM for fine-grained reasoning. A Probability-Informed Visual Enhancement (PVE) objective masks high-frequency tokens during training, steering the Resampler to concentrate on semantically rich regions-just as skilled reader gloss over function words. On eleven in-context learning benchmarks, Vist achieves the same accuracy with 2.3 times fewer tokens, cutting FLOPs by 16% and memory by 50%. This method delivers remarkable results, outperforming the strongest text encoder-based compression method CEPE by 7.6% on average over benchmarks like TriviaQA, NQ, PopQA, NLUI, and CLIN, setting a new standard for token efficiency in LLMs. The project is at https://github.com/CSU-JPG/VIST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。