通过精简训练数据,让视觉语言模型回答更简洁,显著提升推理效率。
Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curation

- 用高质量精简数据训练模型,使其输出更短
- 在接近同等准确率下,推理耗能降低35倍
- 模型越大规模,简洁性带来的准确率提升越明显
推理效率通常通过缩小模型实现:如蒸馏、剪枝、量化和稀疏路由,这些方法虽降低每个词元的计算成本,却忽略输出长度增长的问题。本文提出,简洁性是被忽视的效率杠杆,而预训练数据精简是实现该目标的有效途径。在单图子集MAmmoTH-VL上应用数据精简流程,对比经精简数据、原始数据及外部开源前沿模型训练的14个1B-4B参数模型。在控制输出长度条件下,通过每模型回归分离简洁性与质量,并以每正确答案的TFLOPs衡量代价。结果表明,精简数据训练模型相比最冗长的4B模型(Qwen3.5-4B),在仅损失约1个百分点准确率(0.41 vs 14.58 TFLOPs/正确答案;0.691 vs 0.704平均准确率)的情况下,获得35倍的推理成本优势。同时,在相同输出长度下,准确率较未精简基线提升17.55个百分点,且随模型规模增大(1B时+16.7,4B时+21.2)。无论能力或规模,通用冗长表达均无法带来准确率提升,高阶推理冗余的适用范围也从2B时的8组中的4组缩小至4B时的1组。更关键的是,简洁模型甚至能捕捉到冗长推理模型遗漏的正确答案,说明推理是可被精简的数据目标而非牺牲项。此阶段的推理效率本质是‘每正确答案所耗令牌’问题,而简洁性正是直接作用于此的关键杠杆。
原文摘要 · Abstract (English)
Inference efficiency is typically pursued by shrinking the model: distillation, pruning, quantization, and sparse routing each lower per-token cost while treating token count as fixed. But output length has been inflating, and it is precisely the component the standard toolkit leaves untouched. Here, we argue that brevity is the missing inference-efficiency lever, and that pretraining data curation is a practical way to pull it: a model trained on concise, correct data learns to answer in fewer tokens; i.e. it has a lower Cost-of-Pass. We apply our VLM curation pipeline to the MAmmoTH-VL single-image subset, and compare models trained on our curated data, the standard MAmmoTH-VL data, and external open-weight frontier VLMs. On a controlled 20-evaluation set and 14 VLMs at 1B-4B activated parameters, we hold output length fixed with a per-model regression, separating brevity from quality, and price models in FLOPs per correct answer. Curation buys a 35x Cost-of-Pass advantage over the most verbose 4B comparator (Qwen3.5-4B) within $\sim$1 pp of accuracy (0.41 vs 14.58 TFLOPs per correct answer; 0.691 vs 0.704 mean accuracy). Curation also buys a +17.55-percentage-point matched-length accuracy gain over the uncurated baseline that grows with model scale (from +16.7 pp at 1B to +21.2 pp at 4B). This brevity improvement concedes no quality: generic verbosity buys no accuracy at any capability or scale, and the window where reasoning-structured verbosity still earns its tokens shrinks from 4 of 8 capability groups at 2B to 1 of 8 at 4B. Per example, the concise model even reaches correct answers the verbose reasoning model misses, marking reasoning as a distinct curation target rather than something brevity gives up. Inference efficiency in this regime is a tokens-per-correct problem, and brevity is the lever that targets it directly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。