发现大模型在普通输入下会过度输出,导致成本和能耗飙升。
BenchOverflow: Measuring Overflow in Large Language Models via Plain-Text Prompts
- 用9种纯文本提示策略测试模型输出膨胀问题
- 多数模型在5000令牌预算下出现严重长尾分布
- 轻量提醒可有效抑制过度输出,适合生产部署
我们研究了大语言模型(LLMs)在普通交互中因纯文本提示引发过度输出的现象,称为Overflow。该现象不依赖恶意后缀或策略绕过,却会导致服务成本上升、延迟增加及跨用户性能下降,尤其在大规模请求下更为显著。不必要的输出增加了每请求成本与能耗,放大至整体运营开支与碳足迹。此外,Overflow是共享环境中计算资源被滥用的现实风险。为此,我们提出BenchOverflow,一个不依赖特定模型的基准,包含九种纯文本触发策略,在固定5000新令牌预算下评估九个开源与闭源模型。结果显示输出长度分布普遍右偏且存在重尾。通过容积饱和率(CSR@1k/3k/5k)与经验累积分布函数(ECDF)量化尾部风险;提示内方差与跨模型相关性表明该问题可复现但呈现家族与攻击方式差异。一种轻量级缓解方案——固定简洁提醒——对多数模型的全部策略均有效降低右尾与CSR。研究揭示长度控制是可度量的可靠性、成本与可持续性问题,而非风格偏好。BenchOverflow为跨模型长度控制鲁棒性的标准化比较提供了基础,助力选择低资源浪费的部署方案,并评估防御措施在不牺牲任务表现前提下的有效性。
原文摘要 · Abstract (English)
We investigate a failure mode of large language models (LLMs) in which plain-text prompts elicit excessive outputs, a phenomenon we term Overflow. Unlike jailbreaks or prompt injection, Overflow arises under ordinary interaction settings and can lead to elevated serving cost, latency, and cross-user performance degradation, particularly when scaled across many requests. Beyond usability, the stakes are economic and environmental: unnecessary tokens increase per-request cost and energy consumption, compounding into substantial operational spend and carbon footprint at scale. Moreover, Overflow represents a practical vector for compute amplification and service degradation in shared environments. We introduce BenchOverflow, a model-agnostic benchmark of nine plain-text prompting strategies that amplify output volume without adversarial suffixes or policy circumvention. Using a standardized protocol with a fixed budget of 5000 new tokens, we evaluate nine open- and closed-source models and observe pronounced rightward shifts and heavy tails in length distributions. Cap-saturation rates (CSR@1k/3k/5k) and empirical cumulative distribution functions (ECDFs) quantify tail risk; within-prompt variance and cross-model correlations show that Overflow is broadly reproducible yet heterogeneous across families and attack vectors. A lightweight mitigation-a fixed conciseness reminder-attenuates right tails and lowers CSR for all strategies across the majority of models. Our findings position length control as a measurable reliability, cost, and sustainability concern rather than a stylistic quirk. By enabling standardized comparison of length-control robustness across models, BenchOverflow provides a practical basis for selecting deployments that minimize resource waste and operating expense, and for evaluating defenses that curb compute amplification without eroding task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。