动态调整提示样本数,提升大模型推理效率与准确率。
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse

- 根据输出熵自动确定最优提示样本数。
- 通过语义感知的缓存复用,实现4.64倍加速。
- 适合需要高效推理的复杂任务场景。
多示例上下文学习(Many-Shot In-Context Learning, ICL)已成为激发大语言模型(LLM)推理潜力的重要范式,但现有方法通常采用固定数量的示例,难以适应不同查询的难度差异,导致上下文不足或噪声干扰。此外,长上下文带来的计算与内存开销严重限制了其可行性。为此,我们提出AdapShot,通过动态优化示例数量并利用键值缓存复用实现高效推理。具体而言,设计基于探针的评估机制,利用输出熵判断最优示例数;为避免探测与推理阶段重复预填充计算,引入语义感知的缓存复用策略,并提出解耦与重编码方法解决位置编码不兼容问题,支持缓存项灵活重排。大量实验表明,AdapShot相比当前最优方法DBSA平均性能提升约10%,推理速度提升4.64倍。
原文摘要 · Abstract (English)
Many-Shot In-Context Learning (ICL) has emerged as a promising paradigm, leveraging extensive examples to unlock the reasoning potential of Large Language Models (LLMs). However, existing methods typically rely on a predetermined, fixed number of shots. This static approach often fails to adapt to the varying difficulty of different queries, leading to either insufficient context or interference from noise. Furthermore, the prohibitive computational and memory costs of long contexts severely limit Many-Shot's feasibility. To address the above limitations, we propose AdapShot, which dynamically optimizes shot counts and leverages KV cache reuse for efficient inference. Specifically, we design a probe-based evaluation mechanism that utilizes output entropy to determine the optimal number of shots. To bypass the redundant prefilling computation during both the probing and inference phases, we incorporate a semantics-aware KV cache reuse strategy. Within this reuse strategy, to address positional encoding incompatibilities, we introduce a decoupling and re-encoding method that enables the flexible reordering of cached key-value pairs. Extensive experiments demonstrate that AdapShot achieves an average performance gain of around 10% and a 4.64x speedup compared to state-of-the-art DBSA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。