arXiv:2603.20224cs.CL2026-03被引 25

用每令牌能耗评估大模型推理效率,实现节能与精度的动态平衡。

Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference

  • 提出每令牌能耗指标,替代传统准确率单一衡量标准。
  • 发现小模型加推理策略后能耗可能反升,存在能效权衡。
  • 基于硬件能耗曲线动态控制推理深度,适合绿色AI部署者。

大型语言模型在多种任务中表现优异,但计算和能源成本高昂,尤其在高请求场景下。许多实际应用中,小型语言模型(SLMs)足以完成简单文本生成任务,且通过链式思维(CoT)提示或多数投票等增强推理策略,可逼近大模型性能,同时降低总体计算开销。然而,这些策略也可能引入额外能耗,形成能效权衡。本文以MMLU基准测试为基础,分析小模型在测试时计算策略下的能效表现;同时研究Transformer架构的输入输出令牌动态特性,揭示其导致硬件能耗非线性变化。为推动人工智能研究与物理影响结合,我们提出能量效率指标,包括每令牌能耗,作为传统准确率的补充。此外,我们建议在生成链式思维令牌时进行受控推理,利用硬件能耗曲线动态调节推理深度。该方案整合了能耗感知路由机制,使模型选择与推理策略协同优化,在保障准确率的同时实现可持续的AI部署。

原文摘要 · Abstract (English)

Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks but come with substantial energy and computational costs, particularly in request-heavy scenarios. In many real-world applications, the full scale and capabilities of LLMs are often unnecessary, as Small Language Models (SLMs) can provide accurate responses for simpler text generation tasks. When enhanced with advanced reasoning strategies, such as Chain-of-Thought (CoT) prompting or Majority Voting, SLMs can approach the performance of larger models while reducing overall computational requirements. However, these strategies can also introduce additional energy costs, creating an energy-accuracy trade-off. Our analysis examines these trade-offs in test-time compute strategies for smaller models compared to larger ones, using the MMLU benchmark. Additionally, we explore the input-output token dynamics of transformer architectures, which result in nonlinear hardware energy operation curves for LLMs. To bridge AI research with its physical impact, we propose \textit{energy efficiency metrics}, including Energy-per-Token, as complements to traditional accuracy benchmarks. Beyond model selection, we propose controlled reasoning in CoT token generation, using operating curves to regulate reasoning depth dynamically. This vision integrates a energy-aware routing mechanism, ensuring that model selection and inference strategies balance accuracy for sustainable AI deployment.

能效评估小模型推理优化绿色AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。