通过子词典加速大模型推理，既快又保证结果准确
CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference
- 用几何边界筛选每步输出的候选词，只算重要部分
- 实测速度提升显著，错误率低于1%且可验证
- 适合需要高速高可靠推理的场景，如医疗、金融
大语言模型推理时因词汇表过大导致计算瓶颈。本文提出CSV-Decode，利用几何上界构建每步的微型子词汇表，实现高效稀疏计算，并提供双重正确性保障：精确的top-k认证与ε-可证的softmax近似。方法离线聚类词嵌入，通过中心点加半径约束识别可安全忽略的词。我们实现了完整系统，包含稀疏GEMV内核、多GPU分片及CUDA Graph优化。实验表明，在保持分布一致性与低回退率的前提下，相比全词汇表解码有显著提速。代码已开源。
原文摘要 · Abstract (English)
Large language models face significant computational bottlenecks during inference due to the expensive output layer computation over large vocabularies. We present CSV-Decode, a novel approach that uses geometric upper bounds to construct small sub-vocabularies for each decoding step, enabling efficient sparse computation while maintaining dual correctness guarantees: exact top-$k$ certification and $\varepsilon$-certified softmax approximations. Our method clusters vocabulary embeddings offline and uses centroid-plus-radius bounds to identify which tokens can be safely omitted from computation. We provide a complete system implementation with sparse GEMV kernels, multi-GPU sharding, and CUDA Graph optimization. Experimental results demonstrate significant speedup over full vocabulary decoding while maintaining distributional guarantees and low fallback rates. Our code implementation available at \href{https://github.com/FastLM/CSV-Decode}{https://github.com/FastLM/CSV-Decode}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。