用小模型实现低成本实时任务路由,效果接近大模型。
Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment
- 用30亿参数小模型在相同硬件上做任务分类,精度达78.3%。
- 3B模型在延迟与准确率间表现最佳,零边际成本且响应快于1秒。
- 适合关注推理效率与部署成本的系统设计者参考。
在推理时选择合适的模型——路由问题——需综合优化输出质量、成本、延迟和治理约束。现有方法依赖昂贵且高延迟的LLM分类器或偏好训练的路由器,将多目标优化简化为单一质量预测。我们主张,当前的小语言模型(SLMs,1-4B参数)已具备足够推理能力,可实现亚秒级、零边际成本、自托管的任务分类,使路由决策对推理预算影响微乎其微。通过两项研究验证该假设:研究1在相同Azure T4硬件、服务栈、量化方式及60个样本的语料库下,对比Phi-3.5-mini、Qwen2.5-1.5B和Qwen-2.5-3B在六类任务上的表现,结果显示Qwen-2.5-3B取得最高精确匹配准确率(0.783),具有最强的延迟-准确率权衡能力,并在所有六类任务中均实现非零准确率。研究2为预注册的四臂随机实验,在合成流量下有效样本量每组60例,比较Phi-4-mini、Qwen-2.5-3B、DeepSeek-V3与无路由控制组。DeepSeek-V3达到最高准确率(0.830),但未通过预注册的P95延迟门限(2,295毫秒);Qwen-2.5-3B在自托管模型中呈帕累托最优(准确率0.793,中位延迟988毫秒,边际成本为0)。无模型满足独立可行性标准(≥0.85准确率,≤2,000毫秒P95)。小模型路由的成本与延迟要求已满足,但6-8个百分点的准确率差距以及正确分类是否提升下游输出质量仍是投入生产前的未解难题。
原文摘要 · Abstract (English)
Selecting the appropriate model at inference time -- the routing problem -- requires jointly optimizing output quality, cost, latency, and governance constraints. Existing approaches delegate this decision to LLM-based classifiers or preference-trained routers that are themselves costly and high-latency, reducing a multi-objective optimization to single-dimensional quality prediction. We argue that small language models (SLMs, 1-4B parameters) have now achieved sufficient reasoning capability for sub-second, zero-marginal-cost, self-hosted task classification, potentially making the routing decision negligible in the inference budget. We test this thesis on a six-label taxonomy through two studies. Study 1 is a harmonized offline benchmark of Phi-3.5-mini, Qwen2.5-1.5B, and Qwen-2.5-3B on identical Azure T4 hardware, serving stack, quantization, and a fixed 60-case corpus. Qwen-2.5-3B achieves the best exact-match accuracy (0.783), the strongest latency-accuracy tradeoff, and the only nonzero accuracy on all six task families. Study 2 is a pre-registered four-arm randomized experiment under synthetic traffic with an effective sample size of 60 unique cases per arm, comparing Phi-4-mini, Qwen-2.5-3B, and DeepSeek-V3 against a no-routing control. DeepSeek-V3 attains the highest accuracy (0.830) but fails the pre-registered P95 latency gate (2,295 ms); Qwen-2.5-3B is Pareto-dominant among self-hosted models (0.793 accuracy, 988 ms median, $0 marginal cost). No model meets the standalone viability criterion (>=0.85 accuracy, <=2,000 ms P95). The cost and latency prerequisites for SLM-based routing are met; the accuracy gap of 6-8 percentage points and the untested question of whether correct classification translates to downstream output quality bound the remaining distance to production viability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。