提出动态集成草案方法，提升视觉语言模型推理速度与稳定性。
TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs
- 测试时动态集成多个草案，利用历史真实输出偏差调整权重。
- 平均加速1.74倍，比单个草案方法快5%，且无需训练。
- 无需额外开销，可兼容多种验证与草案生成方式，即插即用。
推测解码（SD）通过快速生成草案令牌并并行验证，显著加速大语言模型推理。然而，该技术在处理图像与文本的大型视觉语言模型（LVLMs）中仍缺乏研究。我们针对11个不同数据集、多种输入场景，评估了小型草案模型的现有推理方法，发现性能随场景波动明显。为此，提出测试时自适应批处理集成草案（TABED），通过利用SD设置中可用的历史真实输出偏差，动态集成批量推断获得的多个草案。该动态集成方法在保持参数共享、计算成本几乎为零的前提下，实现平均1.74倍的鲁棒墙时加速，较自回归解码显著提升，且比单一草案方法提高5%。凭借即插即用特性，进一步融合先进验证与替代草案生成方法。代码与定制训练模型已公开于https://github.com/furiosa-ai/TABED。
原文摘要 · Abstract (English)
Speculative decoding (SD) has proven effective for accelerating LLM inference by quickly generating draft tokens and verifying them in parallel. However, SD remains largely unexplored for Large Vision-Language Models (LVLMs), which extend LLMs to process both image and text prompts. To address this gap, we benchmark existing inference methods with small draft models on 11 datasets across diverse input scenarios and observe scenario-specific performance fluctuations. Motivated by these findings, we propose Test-time Adaptive Batched Ensemble Drafting (TABED), which dynamically ensembles multiple drafts obtained via batch inference by leveraging deviations from past ground truths available in the SD setting. The dynamic ensemble method achieves an average robust walltime speedup of 1.74x over autoregressive decoding and a 5% improvement over single drafting methods, while remaining training-free and keeping ensembling costs negligible through parameter sharing. With its plug-and-play compatibility, we further enhance TABED by integrating advanced verification and alternative drafting methods. Code and custom-trained models are available at https://github.com/furiosa-ai/TABED.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。