解决大模型任务失败重试导致的令牌成本虚高问题。
Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems

- 通过系统测量不同模型和任务的令牌膨胀率,发现最高达4.25倍。
- 提出基于推理过程熵的难度预判方法,预测准确率达0.887 AUROC。
- 采用按语义换算率动态选型+失败链重置策略,节省31%令牌。
当语言模型首次无法回答问题时,代理系统会重试,每次消耗额外令牌。这种重试开销造成模型单令牌定价与实际工作流成本之间的差距,称为‘令牌膨胀’,定义为真实成本与单次调用成本之比。现有系统如FrugalGPT仅依据单次成本路由,对复杂任务的成本低估超过2倍。本文提出InflationAgent,一个四阶段路由器:(1)系统性测量不同模型层级与任务类型下的令牌膨胀,发现7B模型在多跳问答任务中膨胀高达4.25倍;(2)提出思维链分支熵(CBE),一种仅依赖本地推理的预执行难度信号,可实现0.887 AUROC的高膨胀预测;(3)基于语义换算率(SER)选择模型,即预期准确率除以预测真实成本,并引入新链式升级策略,在失败后丢弃原推理链再调用更强模型。在固定预算下,GSM8K任务中InflationAgent达成94.7%准确率,优于FrugalGPT的91.0%,同时减少31%令牌消耗;实验表明将失败链转发至GPT-4o可使准确率下降最多34.8个百分点,验证了新链设计的有效性。
原文摘要 · Abstract (English)
When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time. This retry overhead creates a gap between what a model's per-token price implies and what a full workflow actually costs. We call this gap \emph{token inflation} and define it as the ratio of true workflow cost to single-call cost. Systems like FrugalGPT route based on the latter, which can underestimate real cost by more than $2\times$ on difficult tasks. We address this with InflationAgent, a four-stage router that (1) measures token inflation systematically across model tiers and task types, finding inflation as high as $4.25\times$ for a 7B model on multi-hop question answering; (2) introduces CoT Branching Entropy (CBE), a pre-execution difficulty signal computed entirely from local inference, which predicts high inflation with AUROC 0.887; and (3) selects models by maximizing a Semantic Exchange Rate (SER) that divides expected accuracy by predicted true cost, with a fresh-escalation policy that discards failed chains before routing to a stronger model. On GSM8K under a fixed budget, InflationAgent achieves 94.7\% accuracy versus 91.0\% for FrugalGPT while using 31\% fewer tokens, and we show that forwarding a failed reasoning chain to GPT-4o reduces its accuracy by up to 34.8 percentage points, validating the fresh-escalation design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。