通过分步复用与轻量验证，显著加速大模型推理并保证输出正确性。
StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving
- 将输出拆分为有序步骤，按步复用并仅重生成失败部分。
- 在数学和JSON任务中，平均延迟降至0.67秒，正确率提升至100%。
- 适合需要结构化输出或高可靠性的部署场景。
我们针对重复请求具有共同解结构但局部约束不同的大模型服务场景（如输出格式、变量名或数值常量差异）。现有缓存方法要么复用完整响应（语义缓存），要么复用模型内部的键值状态，前者对局部修改敏感，后者强耦合特定后端。本文提出StepCache，一种无后端依赖的分步复用层：将输出分段为有序步骤，检索最匹配的缓存请求，使用轻量级任务感知检查验证步骤，并通过选择性修补仅重生成失败区域。该方案支持严格的结构化输出控制，包括单步提取、必填字段约束及一次修复机制，还提供保守跳过复用的回退策略。对于线性方程问题，通过有界修复循环将验证转为纠错，确保当后端模型失败时仍能保持正确性。在仅用CPU的扰动密集型微基准测试中，平均延迟从2.13秒降至0.67秒，中位延迟从2.42秒降至0.01秒，p95延迟从3.38秒降至3.30秒；总词元数从36.1k降至27.3k；端到端正确率从72.5%提升至100%。跨请求统计显示，79.7%请求走纯复用路径，5.4%需修补，14.9%触发跳过复用。
原文摘要 · Abstract (English)
We address LLM serving workloads where repeated requests share a common solution structure but differ in localized constraints, such as output schema, variable names, or numeric constants. Prior caching approaches typically reuse either full responses (semantic caching) or model-internal KV/prefix states, which are respectively brittle under partial changes or tightly coupled to specific backends. We present StepCache, a backend-agnostic step-level reuse layer that segments outputs into ordered steps, retrieves the best-matching cached request, verifies steps using lightweight task-aware checks, and regenerates only failing regions via selective patching. StepCache additionally supports strict structured-output enforcement for JSON, including single-step extraction, required-key constraints, and one-shot repair, as well as conservative skip-reuse fallbacks for semantic changes. For linear equations, StepCache promotes verification into correction via a bounded repair loop with a deterministic fallback that guarantees correctness when the backend model fails. In a CPU-only perturbation-heavy micro-benchmark on math and JSON variants, averaged over three seeds, StepCache reduces mean latency from 2.13 s to 0.67 s, median latency from 2.42 s to 0.01 s, and p95 latency from 3.38 s to 3.30 s. It also reduces total token usage from 36.1k to 27.3k and improves end-to-end correctness from 72.5% to 100% under task-specific checks and a stitched-output integrity check. Across requests, 79.7% take the reuse-only fast path, 5.4% require patching, and 14.9% trigger skip-reuse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。