破解商业大模型的隐形推理计费,防止服务商虚报token数量。
CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs

- 通过嵌入指纹构建可验证哈希树,审计隐藏推理过程的token数量。
- 用语义相关性检测识别伪造推理内容,成功率达94.7%。
- 适合关注大模型服务透明度与成本控制的研究者和开发者。
随着后训练技术的发展,大型语言模型(LLMs)逐渐具备结构化多步推理能力,通常通过强化学习优化。这些增强推理能力的模型在复杂任务上表现优于普通LLM,现已成为许多商业LLM API的核心。然而,为保护专有行为并减少冗余输出,服务提供商通常隐藏推理过程,仅返回最终答案。这种不透明性导致关键透明度缺失:用户需为不可见的推理token付费,而这些token常占总成本的绝大部分,却无法验证其真实性。这引发了计费膨胀风险,如服务商可能虚报使用量或注入低质量合成token以提高费用。为此,我们提出CoIn,一种验证框架,用于审计隐藏token的数量与语义有效性。CoIn通过令牌嵌入指纹构建可验证哈希树来检查数量,并利用嵌入相关性匹配检测伪造推理内容。实验表明,当作为可信第三方审计者部署时,CoIn能有效检测计费膨胀,成功率最高达94.7%,显著提升不透明大模型服务的计费透明度。数据集与代码已公开于https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn。
原文摘要 · Abstract (English)
As post-training techniques evolve, large language models (LLMs) are increasingly augmented with structured multi-step reasoning abilities, often optimized through reinforcement learning. These reasoning-enhanced models outperform standard LLMs on complex tasks and now underpin many commercial LLM APIs. However, to protect proprietary behavior and reduce verbosity, providers typically conceal the reasoning traces while returning only the final answer. This opacity introduces a critical transparency gap: users are billed for invisible reasoning tokens, which often account for the majority of the cost, yet have no means to verify their authenticity. This opens the door to token count inflation, where providers may overreport token usage or inject synthetic, low-effort tokens to inflate charges. To address this issue, we propose CoIn, a verification framework that audits both the quantity and semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from token embedding fingerprints to check token counts, and uses embedding-based relevance matching to detect fabricated reasoning content. Experiments demonstrate that CoIn, when deployed as a trusted third-party auditor, can effectively detect token count inflation with a success rate reaching up to 94.7%, showing the strong ability to restore billing transparency in opaque LLM services. The dataset and code are available at https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。