通过动态调度推理分支，显著降低大模型推理的耗时与资源开销。
Intra-request branch orchestration for efficient LLM reasoning
- 基于层激活预测分支正确性，动态决定终止、复制或继续分支。
- 在保持准确率不变的前提下，减少42%-63%的令牌消耗。
- 适合高并发场景，尤其对延迟敏感的实时推理系统有明显提升。
大型语言模型（LLMs）越来越多地依赖推理时的思维链和多分支推理算法来提升复杂任务的准确性，但这些方法显著增加了令牌使用量和请求延迟。现有工作主要关注减少令牌消耗，常以牺牲准确性为代价，忽略了其他延迟因素。本文提出DUCHESS，一种通过预测引导的请求内分支调度机制，在不损失准确性的前提下降低推理成本与延迟。DUCHESS采用轻量级线性探测模型，基于LLM层激活估计分支正确性，并据此决策是否终止、复制或继续分支。处理多个请求时，它还能根据提示预估难度，优先调度较简单任务。在三个推理基准测试中，DUCHESS始终优于自一致性方法，在相同准确率下减少42%-63%的令牌用量。在vLLM服务中，其平均、中位及尾部延迟分别降低57%-81%、58%-85%和52%-84%，在难度感知调度下于高请求率时表现更优。
原文摘要 · Abstract (English)
Large Language Models (LLMs) increasingly rely on inference-time reasoning algorithms such as chain-of-thought and multi-branch reasoning to improve accuracy on complex tasks. These methods, however, substantially increase token usage and per-request latency. Prior work has largely focused on reducing token usage, often at the expense of accuracy, while overlooking other latency factors. We present DUCHESS, an LLM serving system that reduces cost and latency without sacrificing accuracy through intra-request branch orchestration guided by predictions. DUCHESS employs a lightweight linear probing model over LLM layer activations to estimate branch correctness, and its orchestration policy decides whether to terminate, duplicate, or continue a branch. When handling multiple requests, DUCHESS further reduces latency by prioritizing easier reasoning tasks when complexity can be estimated from the prompt. Experiments on three reasoning benchmarks show that DUCHESS consistently improves the token-accuracy Pareto frontier, reducing token usage by 42-63% at matched accuracy compared to self-consistency. In serving with vLLM, DUCHESS reduces mean, median, and tail latencies by 57-81%, 58-85%, and 52-84% with First-Come-First-Served scheduling, and achieves additional gains under difficulty-aware scheduling at higher request rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。