用语义编码替代原始缓存传输，显著提速大模型推理
Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching
- 用低秩子空间重建多数层缓存，减少传输开销
- 在稀疏层预测输入并修正误差，控制质量下降
- 适合带宽受限场景，兼顾速度与生成质量
去中心化推理缓解了大语言模型推理中的内存瓶颈，但带来了严重的通信瓶颈：高维键值（KV）缓存传输常成为首字延迟（TTFT）的主要负担。此外，在异构模型间复用缓存（如基础模型与微调版本）会导致语义错位，错误随层数累积，降低生成质量。我们提出语义缓存蒸馏（SCD），一种受损失约束的框架，将原始KV传输替换为紧凑的语义编码。SCD通过两种机制解决该问题：(1) 复用——从低秩子空间重构大部分层以最小化传输成本；(2) 补丁——在稀疏过渡层预测归一化输入，截断误差传播。实验表明，SCD相比理想消费者预填充可实现最高2.65倍的TTFT加速，在带宽受限环境下优于量化和选择性重计算基线，且生成质量保持在理想值5% F1以内。
原文摘要 · Abstract (English)
Disaggregated serving alleviates memory bottlenecks in Large Language Model (LLM) inference but creates a severe communication bottleneck: transmitting high-dimensional Key-Value (KV) caches often dominates time-to-first-token (TTFT). Moreover, reusing caches across heterogeneous models (e.g., base and fine-tuned variants) causes semantic misalignment that accumulates over layers, degrading generation quality. We propose Semantic Cache Distillation (SCD), a loss-constrained framework that replaces raw KV transmission with compact semantic codes. SCD addresses these challenges via two mechanisms: (1) Reuse, which reconstructs most layers from low-rank subspaces to minimize transfer cost, and (2) Patch, which predicts normalized inputs at sparse transition layers to truncate error propagation. Empirically, SCD delivers up to 2.65 $\times$ TTFT speedup over the oracle consumer prefill and dominates quantization and selective recomputation baselines on the quality--latency Pareto frontier in bandwidth-constrained regimes, while keeping generation quality within 5\% F1 of the oracle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。