提出缓存感知的提示压缩方法，显著降低大模型调用成本。
Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
- 采用无查询依赖的压缩策略，避免破坏缓存命中率
- 在长提示场景下平均节省49%成本，质量损失小于0.05
- 适合生产环境高并发、长上下文任务使用
实际大模型部署中，提示缓存与提示压缩常协同使用以降低成本。现有压缩方法多为基于查询的自适应压缩，导致每次请求都使缓存失效。我们在Anthropic Sonnet 4.6 API上实测发现，其缓存具有两层结构，在3500令牌以下时缓存命中率稳定在约0.83，远低于文献假设的理想值ρ=1.0。成本模型预测并实验验证：在真实命中率下，高压缩比（r≥6）时，无查询依赖压缩优于传统缓存。为此提出缓存感知提示压缩（CAPC），结合无查询压缩与显式缓存控制，并引入层级保持压缩比例约束，防止过度压缩导致前缀进入高速缓存层。在LongBench-v2上，CAPC在16/16配置中均最优，平均相比仅缓存节省49%，相比查询感知压缩节省64%，相比原始输入节省90%，且质量仅下降0.05。在三个生产级任务中验证：企业工具助手（51.7%成本下降，压缩比r=3）；跨代码库知识图谱RAG管道（FastAPI上9.3倍，httpx上2.4倍优于缓存全量）；tau-bench零售基准（50任务），CAPC为最经济方案，奖励与原生一致（均为36/50，p=1.00），而查询感知压缩成本高出40.1%，首次在公开基准上证实交叉模型负收益预测。
原文摘要 · Abstract (English)
Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent). The compression literature has standardized on query-aware methods that produce a different compressed prefix per query, mechanically invalidating the prefix-strict cache on every call. We characterize this cost empirically on Anthropic's Sonnet 4.6 API and find caching is far from the rho=1.0 ideal the literature assumes: Sonnet's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at rho~0.83 across 30-call sessions. Our cost model predicts, and experiments confirm, that under realistic rho, query-aware compression beats naive caching at high compression ratios (r>=6). We propose Cache-Aware Prompt Compression (CAPC), pairing query-agnostic compression with explicit cache_control plus a tier-preserving ratio bound that prevents over-compression from pushing the cached prefix into the hot tier. CAPC is the cheapest strategy in 16/16 configurations on LongBench-v2, with mean savings of 49% over cache-only, 64% over query-aware compression, and 90% over vanilla, at quality within 0.05 of the uncompressed baseline. We validate CAPC on three production workloads: an enterprise tool-using assistant with a 94k-token schema prefix (51.7% cost reduction at r=3); a graphify knowledge-graph RAG pipeline across two codebases (9.3x vs cache-all on FastAPI, 2.4x on httpx); and the public tau-bench retail benchmark (50 tasks), where CAPC is the cheapest of four strategies with reward exactly equal to vanilla (both 36/50, p=1.00) while query-aware compression is the most expensive at +40.1% over vanilla -- the first production confirmation of the crossover model's negative-ROI prediction on a public benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。