把文字转成图像输入，能省一半令牌还保持效果。
Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs
- 将长文本渲染为图片直接输入模型，减少解码器令牌数。
- 在两个基准上实现近半的令牌节省，性能无下降。
- 适合需要降低计算成本的长文本处理场景。
大型语言模型（LLMs）及其多模态变体如今可处理视觉输入，包括文本图像。这引发了一个有趣问题：能否通过将文本作为图像输入来压缩输入，从而减少令牌使用量，同时保持性能？本文表明，视觉文本表示是解码器型LLM的一种实用且出人意料有效的输入压缩方式。我们提出将长文本输入渲染为单张图像并直接提供给模型，显著降低了所需的解码器令牌数量，带来一种新型输入压缩。在两个不同基准RULER（长上下文检索）和CNN/DailyMail（文档摘要）上的实验表明，该文本转图像方法实现了大幅令牌节省（通常接近一半），且任务性能未下降。
原文摘要 · Abstract (English)
Large language models (LLMs) and their multimodal variants can now process visual inputs, including images of text. This raises an intriguing question: can we compress textual inputs by feeding them as images to reduce token usage while preserving performance? In this paper, we show that visual text representations are a practical and surprisingly effective form of input compression for decoder LLMs. We exploit the idea of rendering long text inputs as a single image and provide it directly to the model. This leads to dramatically reduced number of decoder tokens required, offering a new form of input compression. Through experiments on two distinct benchmarks RULER (long-context retrieval) and CNN/DailyMail (document summarization) we demonstrate that this text-as-image method yields substantial token savings (often nearly half) without degrading task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。