arXiv:2602.22261cs.LG2026-02

根据问题复杂度动态切换模型,大幅降低大模型推理能耗。

Sustainable LLM Inference using Context-Aware Model Switching

  • 按查询复杂度选择不同大小模型,结合缓存与智能判断。
  • 能耗降低67.5%,简单请求响应快68%,质量保持93.6%。
  • 适合关注绿色AI、部署优化的工程师和研究者。

大语言模型在众多AI应用中占据核心地位,但其日益增长的能耗引发严重的可持续性问题。当前部署普遍存在‘一刀切’推理策略,几乎所有请求都由同一大型模型处理,无论任务难易,造成大量不必要的能源浪费。为此,我们提出一种上下文感知的模型切换方法,根据查询复杂度动态选择合适语言模型。该系统结合重复查询缓存、基于规则的复杂度评分实现快速可解释决策、机器学习分类捕捉语义意图,以及用户自适应组件,持续学习交互模式。在真实对话负载下,使用Gemma3 1B、Gemma3 4B和Qwen3 4B三个开源模型(计算成本各异)进行评估,通过NVML GPU功耗监测测量能耗,同时记录响应延迟、路由准确率及输出质量(BERTScore F1)。结果表明,相比始终使用最大模型,该方法可降低高达67.5%的能耗,同时保持93.6%的响应质量;简单查询的响应时间也显著提升约68%。这证明模型切换推理为更节能、可持续的AI系统提供了切实可行且可扩展的路径。

原文摘要 · Abstract (English)

Large language models have become central to many AI applications, but their growing energy consumption raises serious sustainability concerns. A key limitation in current AI deployments is the reliance on a one-size-fits-all inference strategy where most systems route every request to the same large model, regardless of task complexity, leading to substantial and unnecessary energy waste. To address this issue, we propose a context-aware model switching approach that dynamically selects an appropriate language model based on query complexity. The proposed system uses a Context-Aware Model Switching for Energy-Efficient LLM Inference that combines caching for repeated queries, rulebased complexity scoring for fast and explainable decisions, machine learning classification to capture semantic intent, and a user-adaptive component that learns from interaction patterns over time. The proposed architecture was evaluated using real conversation workloads and three open-source language models (Gemma3 1B, Gemma3 4B and Qwen3 4B) with different computational costs, measuring energy consumption (via NVML GPU power telemetry), response latency, routing accuracy, and output quality (BERTScore F1) to reflect real-world usage conditions. Experimental results show that the model switching approach can reduce energy consumption by up to 67.5% compared to always using the largest model while maintaining a response quality of 93.6%. In addition, the response time for simple queries also improved significantly by approximately 68%. These results show that model switching inference offers a practical and scalable path toward more energy-efficient and sustainable AI systems, demonstrating that significant efficiency gains can be achieved without major sacrifices in response quality.

大模型优化绿色AI推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。