利用缓存的键值对实现高效推理与采样，无需额外计算
Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoning
- 将KV缓存作为轻量级表征，替代完整隐藏状态
- 在Llama和Qwen模型上实现比肩或超越专用嵌入的性能
- 支持快速/慢速思维切换，生成效率提升5.7倍且精度损失小
KV缓存通常仅用于加速自回归解码，但其蕴含的上下文信息可免费用于下游任务。本文提出将KV缓存视为轻量级表征，避免重新计算或存储完整隐藏状态。尽管弱于专用嵌入，但基于KV的表征在两项关键任务中表现充分：(i) Chain-of-Embedding，在Llama-3.1-8B-Instruct和Qwen2-7B-Instruct上达到竞争性或更优性能；(ii) Fast/Slow Thinking Switching，在Qwen3-8B和DeepSeek-R1-Distil-Qwen-14B上实现自适应推理，生成效率最高提升5.7倍，精度损失极小。研究确立了KV缓存作为免费、高效的采样与推理基础，为大模型推理中的表征复用开辟新方向。代码已开源。
原文摘要 · Abstract (English)
KV caches, typically used only to speed up autoregressive decoding, encode contextual information that can be reused for downstream tasks at no extra cost. We propose treating the KV cache as a lightweight representation, eliminating the need to recompute or store full hidden states. Despite being weaker than dedicated embeddings, KV-derived representations are shown to be sufficient for two key applications: \textbf{(i) Chain-of-Embedding}, where they achieve competitive or superior performance on Llama-3.1-8B-Instruct and Qwen2-7B-Instruct; and \textbf{(ii) Fast/Slow Thinking Switching}, where they enable adaptive reasoning on Qwen3-8B and DeepSeek-R1-Distil-Qwen-14B, reducing token generation by up to $5.7\times$ with minimal accuracy loss. Our findings establish KV caches as a free, effective substrate for sampling and reasoning, opening new directions for representation reuse in LLM inference. Code: https://github.com/cmd2001/ICLR2026_KV-Embedding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。