用轻量级路由系统实现零样本模型按任务自动选型,兼顾速度、成本与质量。
SCX Router: Streaming Zero-Shot Model Selection with a Decoder-KV Classifier and a Real-World Task Ontology

- 基于解码器-KV分类器的非自回归路由机制,实时评分模型适配度。
- 在1000个任务上达到0.707综合得分,优于最强固定模型的0.696。
- 支持自定义零样本标签,适合需要动态模型调度的应用场景。
大型语言模型的快速普及和应用多样性带来了优化机会:为每个任务选择最合适的模型,同时优化速度、成本与质量。然而,推理端点在质量、价格、延迟、上下文支持、工具使用、领域专长和推理行为方面差异显著,手动规则难以维护且难以持续获得理想的权衡。我们提出 outer{},一个基于GLiClass的轻量级路由器,无需自回归生成即可为每个推理时的模型标签分配适用性分数。发布的0.6B参数检查点结合Qwen3解码器与浅层双向评分器,其解码器-KV执行路径在会话中保留纯文本键值缓存,仅编码新对话轮次,并评估临时候选标签令牌而不将其加入持久缓存。同一检查点还可预测任务类型、难度、推理模式和预期输出长度,支持自定义零样本标签。我们构建了包含23类、115种任务类型、345个可路由子类型、1,173个合成示例及30个领域的任务本体。利用该结构,生成15万条验证评分任务和1.5万条开放任务。在此基础上训练Qwen3解码器,同时显式分离请求预测与任务属性策略(如资格、成本、缓存复用、安全性和主权)。在六个LiveBench子集上,路由器表现优于平均候选模型;在选定的1,000任务子集上,综合得分达0.707,高于最强固定模型的0.696,且收益随基准不同而变化。
原文摘要 · Abstract (English)
The rapid proliferation of large language models (LLMs) and the growing diversity of their applications presents a unique optimization opportunity: selecting the right model for the task, while optimizing for speed, cost, and quality at a per-task level. However, inference endpoints can vary widely in quality, price, latency, context support, tool use, domain expertise, and reasoning behavior. This heterogeneity makes manual heuristics difficult to maintain and unlikely to achieve consistently favorable speed--cost--quality trade-offs on their own. We introduce \router{}, a lightweight GLiClass-based router that assigns a suitability score to each inference-time model label without autoregressive generation. The released 0.6B-parameter checkpoint combines a Qwen3 decoder with a shallow bidirectional scorer. Its decoder-KV execution path preserves a text-only key--value cache across a session, encodes only new dialogue turns, and evaluates transient candidate-label tokens without adding them to the persistent cache. The same checkpoint also predicts task type, difficulty, reasoning mode, and expected output length, and supports custom zero-shot labels. For task generation, we construct a task ontology with 23 families, 115 task types, 345 routable subtypes, 1,173 synthetic examples, and an orthogonal axis of 30 domains. Using this structure, we generate 150,000 verifier-scored tasks and 15,000 open-ended tasks. We then train the Qwen3 decoder on these tasks, while explicitly separating learned request prediction from per-task policies for attributes such as eligibility, cost, cache reuse, safety, and sovereignty. Across six LiveBench subsets, the router outperforms the mean candidate; on the selected 1,000-task subset, it achieves an aggregate top-1 score of 0.707 versus 0.696 for the strongest fixed model, with benchmark-dependent gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。