arXiv:2607.20475cs.AIcs.CL2026-07

统一加速大模型采样与推测验证,支持动态请求和高效图执行。

SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification

论文配图:SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification
图 1 · 摘自论文原文
  • 将采样全流程融合为可动态调度的统一核函数,支持批量处理。
  • 在混合推测解码场景下最高提速16倍,关键算法提升10倍效率。
  • 适合需要灵活批处理的大规模推理服务部署,兼容CUDA Graph。

大模型推理中的采样涉及大量对数概率处理、词元选择和验证操作。现有实现要么仅加速部分流程,依赖多次内核调用,或假设批量内采样行为一致,难以支持动态服务负载且无法高效使用CUDA Graph。本文提出SonicSampler,一套统一的分块感知Triton内核,将完整的采样流程垂直融合为固定的工作负载感知执行模型。其支持动态每请求采样行为,包括语法约束解码、重复惩罚、频率/存在惩罚、对数偏置、温度缩放、top-k/top-p/min-p过滤以及推测验证——在同一内核中完成批量处理,同时保持完全的CUDA Graph兼容性。核心是新型分层两级top-k算法,在低熵输出结构下实现大规模词表高效选择,相比基准最高提速10倍。在异构推测解码工作负载上,SonicSampler相较当前最优方案最高提速16倍,同时维持灵活批处理能力。

原文摘要 · Abstract (English)

Sampling in LLM inference comprises a combinatorial set of logit processing, token selection, and verification operations for speculative decoding. However, existing implementations either accelerate only subsets of this pipeline, rely on multiple kernel launches, or assume homogeneous sampling behavior across a batch, limiting support for dynamic serving workloads and preventing efficient CUDA Graph execution. We present $\textbf{SonicSampler}$, a unified suite of tile-aware Triton kernels that vertically fuses the complete sampling pipeline into a fixed, workload-aware execution model. Our kernels support dynamic per-request sampling behaviors, including grammar-constrained decoding, repetition, frequency and presence penalties, logit bias, temperature scaling, top-$k$ / top-$p$ / min-$p$ filtering, and speculative verification - within a single batched kernel while remaining fully CUDA Graph-compatible. Central to our approach is a novel hierarchical two-stage top-$k$ algorithm that achieves up to $\textbf{10x speedup}$ over competitive baselines and exploits the low-entropy structure of LLM outputs to enable efficient selection over large vocabularies. Across heterogeneous speculative decoding workloads, SonicSampler achieves up to $\textbf{16x speedup}$ over state-of-the-art baselines while preserving flexible batched execution.

大模型推理采样加速CUDA GraphTriton

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。