arXiv:2502.04428cs.CLcs.AI2025-02被引 34

用不确定性判断是否调用更强模型,提升移动端小模型的可靠性。

Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization

  • 根据模型输出置信度决定是否切换到更强模型
  • 不同小模型和方法下置信度分布差异大,影响路由效果
  • 通过校准数据提升新任务泛化能力,无需额外数据

大型语言模型(LLMs)正越来越多地部署在边缘设备上。为提高效率,常采用推理延迟低、能耗小的小型语言模型(SLMs)。但其在处理复杂任务时易产生错误结果。一种有效方案是基于不确定性的SLM路由:当小模型输出低置信度时,将其高风险请求转发至更强的LLM,遵循“若无把握,寻求更强支持”的原则以增强可靠性。然而依赖更强模型会增加调用成本。如何在效率与效果间取得平衡仍是关键挑战。此外,该策略在新数据集上的泛化能力尚未充分探索。本文在超过1500种设置下系统研究了从SLM到LLM的不确定性驱动路由的基准测试与泛化问题。发现:第一,不同不确定性量化(UQ)方法中,不确定性与正确性的一致性显著影响路由性能;第二,不确定性分布更受具体SLM和选择的UQ方法影响,而非下游数据本身。基于此,我们提出校准数据构建指令流程,并开源了一个保留集以提升新场景下的路由泛化能力。实验表明,仅使用校准数据即可有效提升路由性能,无需新增数据。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly deployed and democratized on edge devices. To improve the efficiency of on-device deployment, small language models (SLMs) are often adopted due to their efficient decoding latency and reduced energy consumption. However, these SLMs often generate inaccurate responses when handling complex queries. One promising solution is uncertainty-based SLM routing, offloading high-stakes queries to stronger LLMs when resulting in low-confidence responses on SLM. This follows the principle of "If you lack confidence, seek stronger support" to enhance reliability. Relying on more powerful LLMs is yet effective but increases invocation costs. Therefore, striking a routing balance between efficiency and efficacy remains a critical challenge. Additionally, efficiently generalizing the routing strategy to new datasets remains under-explored. In this paper, we conduct a comprehensive investigation into benchmarking and generalization of uncertainty-driven routing strategies from SLMs to LLMs over 1500+ settings. Our findings highlight: First, uncertainty-correctness alignment in different uncertainty quantification (UQ) methods significantly impacts routing performance. Second, uncertainty distributions depend more on both the specific SLM and the chosen UQ method, rather than downstream data. Building on the insight, we propose a calibration data construction instruction pipeline and open-source a constructed hold-out set to enhance routing generalization on new downstream scenarios. The experimental results indicate calibration data effectively bootstraps routing performance without any new data.

边缘计算模型路由不确定性量化小模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。