用预计算的键值缓存零成本传递知识，还能实现模型行为调控。
Knowledge Packs: Zero-Token Knowledge Delivery via KV Cache Injection
- 通过预存键值缓存替代传统RAG，实现零令牌开销的知识注入。
- 在700个问题上与RAG完全一致，最高节省95%令牌，正确格式下无偏差。
- 无需训练即可同时实现知识传递和行为微调，适合高效部署场景。
RAG会消耗大量令牌。我们提出知识包：预先计算好的键值缓存，可零令牌成本传递知识。对于因果Transformer，对文本F的前向传播所得的KV缓存，与联合处理F+q的输出完全一致——这源于因果掩码的性质。该等价性精确但脆弱：错误的聊天模板格式会导致6-7个百分点的性能下降，这可能解释了此前声称KV优于RAG的现象。在正确格式下，Qwen3-8B与Llama-3.1-8B在700个问题上无任何偏差，最多节省95%令牌。该KV接口还支持RAG无法实现的行为调控：因RoPE仅旋转键而保留值，对缓存值施加对比性扰动可引导模型行为，而键运算则破坏连贯性。该效应位于中层（33%-66%），独立方向近乎正交（cos≈0），可叠加；知识与调控通道可在α≤0.7下同时运行且互不干扰。无需训练，不修改权重。
原文摘要 · Abstract (English)
RAG wastes tokens. We propose Knowledge Packs: pre-computed KV caches that deliver the same knowledge at zero token cost. For causal transformers, the KV cache from a forward pass on text F is identical to what a joint pass on F+q would produce - this follows directly from the causal mask. The equivalence is exact but fragile: wrong chat template formatting causes 6-7pp degradation, which we believe explains prior claims of KV outperforming RAG. With correct formatting: zero divergences across 700 questions on Qwen3-8B and Llama-3.1-8B, up to 95% token savings. The KV interface also enables behavioral steering that RAG cannot do. Because RoPE rotates keys but leaves values untouched, contrastive deltas on cached values can nudge model behavior while key arithmetic destroys coherence. The effect sits in mid-layer values (33-66%), independent directions are nearly orthogonal (cos~0) and compose, and both channels - knowledge and steering - run simultaneously at alpha<=0.7 without interference. No training, no weight modification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。