提出安全查询路由框架，提升大模型在异常输入下的稳定性。
Guarded Query Routing for Large Language Models
- 设计带防护的查询路由机制，融合多种模型对比评估
- WideMLP在准确率88%下实现<4ms响应速度，表现最佳
- 警告：依赖大模型路由效率低，适合对速度敏感场景
查询路由是将用户请求分配到不同大语言模型（LLM）端点的任务，可视为文本分类问题。然而，必须妥善处理分布外（out-of-distribution）查询，如跨领域、多语言或不安全内容。为此，本文提出受保护的查询路由问题，并发布GQR-Bench基准测试集（以Python包gqr形式开源），涵盖法律、金融、医疗三个目标领域及七组数据集，用于评估模型对分布外查询的鲁棒性。通过该基准，我们对比了基于LLM的路由方法（GPT-4o-mini、Llama-3.2-3B、Llama-3.1-8B）、标准守卫方案（LlamaGuard、NVIDIA NeMo Guardrails）、连续词袋分类器（WideMLP、fastText）以及传统机器学习模型（SVM、XGBoost）。结果表明，增强分布外检测能力的WideMLP在准确率88%与响应时间<4ms间取得最佳平衡；基于嵌入的fastText以<1ms速度达到80%准确率，表现极快；而大模型虽最高准确率达91%，但延迟较高（本地Llama-3.1-8B为62ms，远程GPT-4o-mini达669ms）。研究挑战了对大模型自动路由的盲目依赖，为实际应用提供具体建议。源码已公开：https://github.com/williambrach/gqr。
原文摘要 · Abstract (English)
Query routing, the task to route user queries to different large language model (LLM) endpoints, can be considered as a text classification problem. However, out-of-distribution queries must be handled properly, as those could be about unrelated domains, queries in other languages, or even contain unsafe text. Here, we thus study a guarded query routing problem, for which we first introduce the Guarded Query Routing Benchmark (GQR-Bench, released as Python package gqr), covers three exemplary target domains (law, finance, and healthcare), and seven datasets to test robustness against out-of-distribution queries. We then use GQR-Bench to contrast the effectiveness and efficiency of LLM-based routing mechanisms (GPT-4o-mini, Llama-3.2-3B, and Llama-3.1-8B), standard LLM-based guardrail approaches (LlamaGuard and NVIDIA NeMo Guardrails), continuous bag-of-words classifiers (WideMLP, fastText), and traditional machine learning models (SVM, XGBoost). Our results show that WideMLP, enhanced with out-of-domain detection capabilities, yields the best trade-off between accuracy (88%) and speed (<4ms). The embedding-based fastText excels at speed (<1ms) with acceptable accuracy (80%), whereas LLMs yield the highest accuracy (91%) but are comparatively slow (62ms for local Llama-3.1:8B and 669ms for remote GPT-4o-mini calls). Our findings challenge the automatic reliance on LLMs for (guarded) query routing and provide concrete recommendations for practical applications. Source code is available: https://github.com/williambrach/gqr.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。