将文本嵌入图像降低视觉推理成本，最多省91% token开销。
Token-Efficient Multimodal Reasoning via Image Prompt Packaging
- 把结构化文本直接编码进图像，减少模型输入的文本量。
- 在多个数据集上实现35.8%至91.0%的推理成本下降，部分任务精度仍保持领先。
- 适合关注多模态模型部署效率的研究者和工程师参考。
大规模部署大型多模态语言模型受限于基于token的推理成本，但视觉提示策略的成本-性能表现尚未明确。本文提出图像提示打包（IPPg），将结构化文本直接嵌入图像以降低文本token开销，并在五个数据集、三个前沿模型（GPT-4.1、GPT-4o、Claude 3.5 Sonnet）及两类任务（VQA与代码生成）上进行基准测试。我们推导出按token类型分解的代价公式，结果显示IPPg可实现35.8%–91.0%的推理成本降低。尽管文本压缩最高达96%，多数场景下准确率仍具竞争力，但结果高度依赖模型与任务：GPT-4.1在CoSQL上同时获得准确率与成本提升，而Claude 3.5在多个VQA基准上反而增加成本。系统性错误分析构建了失败模式分类：空间推理、非英文输入与字符敏感操作最易出错，而结构化任务受益最大。125种渲染配置的消融实验显示准确率波动达10–30个百分点，确立视觉编码选择是多模态系统设计中的首要变量。
原文摘要 · Abstract (English)
Deploying large multimodal language models at scale is constrained by token-based inference costs, yet the cost-performance behavior of visual prompting strategies remains poorly characterized. We introduce Image Prompt Packaging (IPPg), a prompting paradigm that embeds structured text directly into images to reduce text token overhead, and benchmark it across five datasets, three frontier models (GPT-4.1, GPT-4o, Claude 3.5 Sonnet), and two task families (VQA and code generation). We derive a cost formulation decomposing savings by token type and show IPPg achieves 35.8--91.0\% inference cost reductions. Despite token compression of up to 96\%, accuracy remains competitive in many settings, though outcomes are highly model- and task-dependent: GPT-4.1 achieves simultaneous accuracy and cost gains on CoSQL, while Claude 3.5 incurs cost increases on several VQA benchmarks. Systematic error analysis yields a failure-mode taxonomy: spatial reasoning, non-English inputs, and character-sensitive operations are most vulnerable, while schema-structured tasks benefit most. A 125-configuration rendering ablation reveals accuracy shifts of 10--30 percentage points, establishing visual encoding choices as a first-class variable in multimodal system design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。