根据问题复杂度自动选择是否推理，提升效率与准确率
When to Reason: Semantic Router for vLLM
- 用语义分类判断问题是否需要推理，仅在必要时启用
- MMLU-Pro上准确率提升10.2个百分点，延迟降低47.1%
- 适合追求高效部署的开源大模型服务场景
大型语言模型在引入思维链等推理模式后，准确率显著提升。然而，推理会带来明显的推理延迟和令牌消耗增加，对环境和成本造成影响，且对许多简单问题并不必要。本文提出一种语义路由器，根据查询的推理需求进行分类，并仅在有益时才选择性应用推理。该方法在MMLU-Pro基准上实现10.2个百分点的准确率提升，同时相比直接使用vLLM进行推理，响应延迟降低47.1%，令牌消耗减少48.5%。结果表明，语义路由为开源大模型服务系统中准确率与效率的平衡提供了有效机制。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate substantial accuracy gains when augmented with reasoning modes such as chain-of-thought and inference-time scaling. However, reasoning also incurs significant costs in inference latency and token usage, with environmental and financial impacts, which are unnecessary for many simple prompts. We present a semantic router that classifies queries based on their reasoning requirements and selectively applies reasoning only when beneficial. Our approach achieves a 10.2 percentage point improvement in accuracy on the MMLU-Pro benchmark while reducing response latency by 47.1% and token consumption by 48.5% compared to direct inference with vLLM. These results demonstrate that semantic routing offers an effective mechanism for striking a balance between accuracy and efficiency in open-source LLM serving systems
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。