用快速模型预判大模型输出，降低推理成本并提速。
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
- 在响应层进行推测解码，无需修改模型结构或共享词表。
- 真实场景下节省45.8%成本，响应速度提升1.83倍，通过跳过调用主模型实现。
- 适合需要低成本高效率部署的大模型服务，尤其适用于代理式任务。
我们提出RLM-Cascade，一种代理层系统，通过在响应层面应用推测解码来降低大模型API成本，无需访问模型架构或共享词汇表。一个快速廉价的草稿模型生成候选响应；一个能力较强的验证模型根据轻量级复杂度路由选择接受、增强或完全跳过。在真实世界代理编码任务（Claude Code）中，RLM-Cascade在125个生产请求中实现了88.8%的草稿使用率，相对于直接调用Opus基线，将API成本降低了45.8%。出人意料的是，该代理还减少了端到端延迟：中位响应时间为2,026毫秒，相比Native Opus的3,698毫秒快了1.83倍（p50）。这是因为跳过路径（仅调用DeepSeek，不调用Opus）主导了工作负载分布。生成质量与或超过Opus基线：在20项代码/数学/指令基准测试中通过率达100%，而Native Opus为95%。我们进一步描述了一种基于规则的复杂度路由机制，用于简单代理任务选择跳过路径，并采用混合工具调用策略，在关键模式匹配任务中绕过推测流水线。RLM-Cascade已作为企业级AI基础设施组件投入生产，并开源发布，包含实时指标仪表盘和Prometheus接口。
原文摘要 · Abstract (English)
We present RLM-Cascade, a proxy-layer system that applies speculative decoding at the response level to reduce LLM API costs without requiring model architecture access or a shared vocabulary. A fast, inexpensive draft model generates a candidate response; a capable verify model accepts, enhances, or is bypassed entirely depending on a lightweight complexity router. On a real-world agentic coding workload (Claude Code), RLM-Cascade achieves a draft-use rate of 88.8% across 125 production requests, reducing API cost by 45.8% relative to a direct Opus baseline. Counter-intuitively, the proxy also reduces end-to-end latency: median response time is 2,026 ms versus 3,698 ms for Native Opus -- a 1.83X speedup at p50 -- because the SKIPPED path (DeepSeek only, no Opus call) dominates the workload distribution. Quality matches or exceeds the Opus baseline: 100% pass rate on a 20-task Code/Math/Instruct benchmark versus 95% for Native Opus. We further describe a rule-based complexity router that selects the SKIPPED path for simple agentic turns and a hybrid tool-call strategy that bypasses the speculative pipeline for schema-critical tool-selection turns. RLM-Cascade is deployed in production as an enterprise AI infrastructure component and published as open source with a live metrics dashboard and Prometheus endpoint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。