让大模型推理更快更准，关键在动态调配计算资源。
Faster and Better LLMs via Latency-Aware Test-Time Scaling
- 用分支并行和推测解码实现并发推理，降低延迟。
- 32B模型1分钟内达82.3%准确率，3B模型10秒内达72.4%。
- 适合对响应速度要求高的实际部署场景。
测试时扩展（TTS）在提升大语言模型推理性能方面已被证明有效，但现有研究忽略了其在延迟敏感场景下的效率问题。通过对代表性TTS方法的延迟感知评估，我们发现计算最优的TTS并不总能带来最低延迟。为解决这一问题并实现延迟最优的TTS，我们提出两种优化并发配置的方法：(1) 分支并行，利用多个并发推理分支；(2) 序列并行，通过推测解码实现。通过合理分配计算资源，我们的延迟最优TTS使32B模型在1分钟内达到MATH-500上82.3%的准确率，3B模型在10秒内达到72.4%的准确率。本工作强调了延迟感知TTS的重要性，并展示了其在延迟敏感场景中同时实现速度与精度的能力。
原文摘要 · Abstract (English)
Test-Time Scaling (TTS) has proven effective in improving the performance of Large Language Models (LLMs) during inference. However, existing research has overlooked the efficiency of TTS from a latency-sensitive perspective. Through a latency-aware evaluation of representative TTS methods, we demonstrate that a compute-optimal TTS does not always result in the lowest latency in scenarios where latency is critical. To address this gap and achieve latency-optimal TTS, we propose two key approaches by optimizing the concurrency configurations: (1) branch-wise parallelism, which leverages multiple concurrent inference branches, and (2) sequence-wise parallelism, enabled by speculative decoding. By integrating these two approaches and allocating computational resources properly to each, our latency-optimal TTS enables a 32B model to reach 82.3% accuracy on MATH-500 within 1 minute and a smaller 3B model to achieve 72.4% within 10 seconds. Our work emphasizes the importance of latency-aware TTS and demonstrates its ability to deliver both speed and accuracy in latency-sensitive scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。