让大模型工作流在等待时提前猜测并执行下一步，省钱又省时。
Cost-Aware Speculative Execution for LLM-Agent Workflows: An Integrated Five-Dimension Method
- 提前用预测输入启动下游任务，减少空等时间。
- 按每令牌计费精确计算猜测成本，成功概率用贝叶斯动态更新。
- 适合对延迟和成本敏感的生产级大模型应用，如自动化工具链。
LLM-agent工作流通过串联模型调用与工具执行，但大部分运行时间耗费在等待上游完成上。推测执行可通过使用预测的上游输入提前启动下游操作来回收空闲时间，但每次推测都需真实付费（按令牌计费），且成功率难以估计且随时间漂移。本文提出五维集成方法：（D1）在上游完成前启动下游；（D2）分别以输入和输出费率实时计价推测成本；（D3）提供单一控制旋钮调节延迟与成本权衡；（D4）采用带失败加权成本项与偏好调整阈值的期望值规则决策；（D5）基于依赖类型分类的贝叶斯贝塔-二项分布后验估计成功率。仅对无副作用、幂等或可置于提交屏障后的操作启用推测，错误推测通过重执行回滚，退款但无法撤销不可逆副作用。论文详述运行时机制、规则自限性（上游分叉因子增大时自动收敛）、五阶段校准流程（离线回放、影子、冒烟、在线校准、漂移触发熔断）及覆盖八种生产原型的工作负载适配准则。与四类近期系统（DSP、Speculative Actions v2、Sherlock、B-PASTE）对比显示各维度均有差异，合成验证套件确认了预测决策边界、概率阈值、后验恢复与流式取消行为。
原文摘要 · Abstract (English)
LLM-agent workflows chain model calls and tool invocations, and spend most of their wall-clock time waiting on upstream operations before downstream ones can start. Speculative execution can reclaim that idle time by launching a downstream operation with a predicted upstream input, but here each speculation costs real money (per-token billing) and its success probability is hard to estimate and drifts over time. This paper presents a method organized around five design decisions: (D1) start a downstream operation before its upstream completes; (D2) price each speculation in real dollars at separate input and output rates; (D3) expose a single operator dial for latency versus cost; (D4) decide via an expected-value rule with a failure-weighted cost term and a preference-adjusted threshold; and (D5) estimate the success probability with a Bayesian Beta-Binomial posterior whose prior is keyed to a dependency-type taxonomy. Variants of these ideas appear in recent work; the combination, with every decision logged in dollars, is what is new. The rule fires only on edges passing an admissibility precondition (side-effect-free, idempotent, or stageable behind a commit barrier), since a wrong speculation is rolled back by re-execution, which refunds tokens but cannot un-send an irreversible side effect. We specify the runtime mechanics, a closed-form result that the rule self-limits as the upstream branching factor grows, a five-stage calibration pipeline (offline replay, shadow, canary, online calibration, drift-triggered kill-switch), and a workload-fit rubric over eight production archetypes. Contrast tables against the four closest published systems (DSP, Speculative Actions v2, Sherlock, B-PASTE) show differentiators on every dimension, and a synthetic validation suite confirms the predicted decision boundary, probability threshold, posterior recovery, and streaming-cancellation behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。