付费按令牌数计费,模型可能偷偷多报令牌数骗钱。
Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives
- 用令牌数计费让服务商有动机虚报生成耗时
- 实验证明可隐蔽地多收用户30%以上费用
- 按字符数收费才公平,适合关注成本的开发者
当前主流大语言模型依赖专用硬件和高能耗,云服务普遍采用按输出令牌数收费。本文揭示该机制催生了服务商虚报令牌数的经济激励,而用户无法验证或察觉。尽管透明化生成过程可抑制欺诈,我们仍设计出一种高效启发式算法,使服务商在不被怀疑的前提下显著多收费——且算法开销低于额外收入。实验基于Llama、Gemma、Ministral系列模型及LMSYS Chatbot Arena数据集,证实此漏洞真实存在。研究进一步指出,唯有按字符数线性定价才能消除欺诈动机;即便利润波动,通过简单调整仍可维持原有平均利润率。这为构建更公平的计费体系提供理论依据。
原文摘要 · Abstract (English)
State-of-the-art large language models require specialized hardware and substantial energy to operate. As a consequence, cloud-based services that provide access to large language models have become very popular. In these services, the price users pay for an output provided by a model depends on the number of tokens the model uses to generate it: they pay a fixed price per token. In this work, we show that this pricing mechanism creates a financial incentive for providers to strategize and misreport the (number of) tokens a model used to generate an output, and users cannot prove, or even know, whether a provider is overcharging them. However, we also show that, if an unfaithful provider is obliged to be transparent about the generative process used by the model, misreporting optimally without raising suspicion is hard. Nevertheless, as a proof-of-concept, we develop an efficient heuristic algorithm that allows providers to significantly overcharge users without raising suspicion. Crucially, we demonstrate that the cost of running the algorithm is lower than the additional revenue from overcharging users, highlighting the vulnerability of users under the current pay-per-token pricing mechanism. Further, we show that, to eliminate the financial incentive to strategize, a pricing mechanism must price tokens linearly on their character count. While this makes a provider's profit margin vary across tokens, we introduce a simple prescription under which the provider who adopts such an incentive-compatible pricing mechanism can maintain the average profit margin they had under the pay-per-token pricing mechanism. Along the way, to illustrate and complement our theoretical results, we conduct experiments with several large language models from the $\texttt{Llama}$, $\texttt{Gemma}$ and $\texttt{Ministral}$ families, and input prompts from the LMSYS Chatbot Arena platform.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。