通过奖励协调压缩与生成,减少推理耗时和冗余。
Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

- 根据推理阶段奖励动态调整缓存压缩强度
- 生成 token 减少 37%-65%,延迟降低 2.08x-2.35x
- 适合追求高效推理的部署场景
大型推理模型(LRMs)通过长链式思维(CoT)在复杂任务上表现优异,但其冗长的中间步骤导致严重过思考,显著增加推理成本。键值缓存(KV-cache)压缩是常见缓解方法,但现有面向推理的方法对整个推理轨迹采用统一压缩策略,仅以移除内容量判断压缩效果。我们发现两个关键现象:一是推理状态对上下文丢失的容忍度随轨迹变化,且过程奖励可追踪这一变化——在高奖励步骤删除令牌比随机删除更能保持准确率;二是压缩并非无代价,更小的缓存会导致模型生成更多令牌,部分抵消节省。为此,我们提出 ReCo(奖励协调压缩),一个分步框架:轻量级过程奖励估计算器为每一步打分,驱动三个组件:(1) 奖励自适应的 KV 缓存压缩,在高奖励步骤更激进地压缩,低奖励则保留更多;(2) 奖励带状的反思令牌惩罚,抑制冗余生成;(3) 基于置信度的早期停止,当推理足够可靠时触发。在三个推理模型和六个基准测试中,ReCo 相比完整 CoT 将生成 token 数减少 37%-65%,端到端延迟降低 2.08x-2.35x,同时基本保持准确性。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache. Two observations point the other way. First, a reasoning state's tolerance to context loss varies along the trajectory, and process reward tracks it: deleting tokens at high-reward steps preserves accuracy far better than deleting the same budget at random. Second, compression is not free on the generation side, since a smaller cache leads the model to generate more tokens, partly canceling the saving. Together these motivate coordinating both sides under a single process reward. We propose ReCo (Reward-Coordinated Compression), a step-wise framework in which a lightweight process-reward estimator scores each completed step and drives three components: (1) reward-adaptive KV-cache compression that shrinks the retained cache harder at high-reward steps and less at low-reward ones, (2) a reward-banded penalty on reflection tokens that curbs redundant generation, and (3) confidence-based early stopping that triggers when the reasoning is reliable. Across three reasoning models and six benchmarks, ReCo reduces generated tokens by 37%-65% and end-to-end latency by 2.08x-2.35x over Full CoT, all while largely preserving accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。