用强化学习动态调整大模型生成速度，无需调参
TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding
- 基于多臂老虎机算法自动选择最优生成策略
- 跨模型和数据集测试中速度提升优于现有方法
- 无需训练或调参，可直接接入任意大模型系统
推测解码通过轻量级草稿模型提前生成候选词，再由大型目标模型并行验证，从而加速大语言模型推理。然而，确定最佳草稿长度仍是关键挑战。动态推测解码旨在智能决定草稿数量以实现最大加速。现有方法通常依赖手工设定的敏感阈值（如词元熵），设置成本高且在不同模型与领域间泛化性差。本文提出 TapOut，一种在线、无需训练、即插即用的动态推测策略选择算法，采用元算法根据历史奖励与探索情况，在多个无参动态推测策略间进行选择。我们在多种模型对与数据集上进行了广泛实验，结果表明，TapOut 在无需任何超参数调优的情况下，达到或超越现有成熟动态推测基线的速度提升效果。
原文摘要 · Abstract (English)
Speculative decoding accelerates LLMs by using a lightweight draft model to generate tokens autoregressively before verifying them in parallel with a larger target model. However, determining the optimal number of tokens to draft remains a key challenge limiting the approach's effectiveness. Dynamic speculative decoding aims to intelligently decide how many tokens to draft to achieve maximum speedups. Existing methods often rely on hand-tuned, sensitive thresholds (e.g., token entropy), which are costly to set and generalize poorly across models and domains. We propose TapOut, an online, training-free, plug-and-play algorithm for dynamic speculation policy selection using multi-armed bandits. Our approach employs a meta-algorithm that selects among multiple parameter-free dynamic speculation strategies based on past reward and exploration. We conduct extensive experiments across diverse model pairs and datasets, showing that TapOut achieves competitive or superior speedups compared to well-established dynamic speculation baselines without any hyperparameter tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。