通过置信度动态跳过层,实现更快的自推测解码。
ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding
- 基于置信度自动选择跳过的中间层,构建轻量草稿模型。
- 在多种模型和数据集上实现最高1.4倍的生成加速。
- 无需训练即可适配不同任务,适合追求推理效率的场景。
自推测解码是一种用于大语言模型的推理技术,旨在加速生成过程而不降低输出质量。该方法结合使用紧凑版模型进行快速近似解码(草稿模型)与全模型的选择性重评估。现有方法通过动态学习推理时跳过的层数来构建草稿模型,形成更小的子网络以加快计算速度。然而,基于启发式的方法在选择跳过层时往往更简单且高效。本文提出ConfLayers,一种基于置信度的动态插件式层跳过方法,用于自推测解码中的草稿模型构建。该方法迭代计算各层置信度,根据自适应阈值选择跳过层,评估结果并更新最优选择,直至无进一步改进或达到最大迭代次数。此框架避免了训练层跳过策略的开销与复杂性,能提供更一致的速度-质量权衡,同时保持草稿模型对多样化任务和数据集的适应性。跨不同模型与数据集的性能评估表明,该方法相比原生LLM生成最高可提升1.4倍速度。
原文摘要 · Abstract (English)
Self-speculative decoding is an inference technique for large language models designed to speed up generation without sacrificing output quality. It combines fast, approximate decoding using a compact version of the model as a draft model with selective re-evaluation by the full target model. Some existing methods form the draft model by dynamically learning which layers to skip during inference, effectively creating a smaller subnetwork to speed up computation. However, using heuristic-based approaches to select layers to skip can often be simpler and more effective. In this paper, we propose ConfLayers, a dynamic plug-and-play approach to forming the draft model in self-speculative decoding via confidence-based intermediate layer skipping. The process iteratively computes confidence scores for all layers, selects layers to skip based on an adaptive threshold, evaluates the performance of the resulting set, and updates the best selection until no further improvement is achieved or a maximum number of iterations is reached. This framework avoids the overhead and complexity of training a layer skipping policy and can provide more consistent speed-quality trade-offs while preserving the adaptivity of the draft model to diverse tasks and datasets. The performance evaluation of ConfLayers across different models and datasets shows that our novel approach offers up to 1.4x speedup over vanilla LLM generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。