为轻量级大模型提供精准的能耗与性能评估,避免盲目选型。
Profiling Lightweight Large Language Models
- 基于PTME框架直接测量精度、时间、内存和能耗四维指标。
- 发现参数量等静态指标无法预测模型精度,资源收紧会显著增加耗时。
- 揭示多目标权衡下的最优配置,适合边缘部署选型参考。
轻量级大语言模型(LLM)正越来越多地在个人电脑上本地部署,未来将在资源受限的边缘和移动端发挥更大作用。在这些场景中,能耗、执行时间与内存占用直接影响可用性,但现有评估主要依赖参数量或浮点运算次数(FLOPs)等代理指标,常与任务精度脱节。本文提出一种基于PTME的实验框架,通过硬件级直接测量,联合评估轻量级LLM推理的精度、执行时间、峰值内存使用和能耗。该方法在受控桌面平台上,对代表性轻量级LLM在边缘类资源约束下进行测试,覆盖代码生成、数学推理和多任务理解等基准。结果表明,静态代理指标虽能较好近似推理成本,但无法预测精度;收紧资源约束会提升成本而不影响精度,且对执行时间的影响强于能耗,大模型受罚最重。此外,无单一模型在所有维度占优,帕累托分析揭示了被仅以准确率或效率评估所掩盖的非主导配置,为不同资源条件下模型选择提供实用指导。研究显示,仅按大小、FLOPs、延迟或准确率选型可能误选部署方案;PTME剖析可发现既保精度又降物理开销的优选配置。
原文摘要 · Abstract (English)
Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments. In such settings, energy consumption, execution time, and memory usage directly affect practical usability, yet existing evaluations of LLM efficiency largely rely on proxy descriptors such as parameter count or FLOPs, often decoupled from task precision. This paper introduces a PTME-based experimental framework for the precision-aware profiling of lightweight LLM inference, jointly measuring Precision, execution Time, peak Memory usage, and Energy consumption through direct hardware-level measurements. The methodology is applied to a representative set of lightweight LLMs executed locally under edge-class resource envelopes on a controlled desktop platform, using benchmarks spanning code generation, mathematical reasoning, and multi-task understanding. We find that static proxy descriptors approximate inference cost well but fail to predict precision. Tightening the resource envelope increases cost without affecting precision, amplifying execution time more strongly than energy and penalizing larger models the most. Moreover, no single model dominates across all PTME dimensions, and a Pareto analysis reveals non-dominated configurations that would be hidden by accuracy-only or efficiency-only assessments, providing practical guidance for selecting models under different resource envelopes. These results show that selecting lightweight LLMs by size, FLOPs, latency, or accuracy alone can select the wrong deployment candidate; PTME profiling exposes configurations that preserve useful accuracy at lower physical cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。