剖析边缘大模型推理的多维度性能，揭示隐藏在平均延迟下的关键差异。
Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels
- 构建统一时序框架，分阶段记录推理过程与硬件数据
- 发现量化降低能耗但不单调影响功耗，不同芯片代际利用率意义不同
- 适合关注边缘部署优化的研究者与工程师参考
边缘大模型部署不仅受模型大小和精度影响，还受推理后端、硬件平台、内存流量和功耗管理等因素制约。本文提出Hydra，一个面向边缘系统级芯片（SoC）上大模型推理的通用架构、分阶段工作负载表征框架。Hydra 在HuggingFace Transformers与llama.cpp中植入统一的每提示词时序记录机制，并融合硬件遥测数据，实现对预填充与解码阶段性能、资源利用率和效率的多维分析。我们评估了三代边缘SoC（AGX Xavier、AGX Orin、AGX Thor）、来自七个模型家族的13个指令微调大模型、五种执行格式及输入输出长度敏感性。生成约10.7万条每提示词记录，全部公开。分析显示：仅看总延迟会掩盖关键部署效应；后端结构改变延迟引入位置；量化虽减少内存流量与能耗，但无法单调预测功耗；不同芯片代际下资源利用率与效率需重新解读。通过关联阶段级时序与系统资源指标，Hydra 支持可复现的、分阶段感知的边缘大模型推理表征。代码与原始轨迹数据开源：https://github.com/amirtaherin/hydra
原文摘要 · Abstract (English)
Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a common-schema, phase-aware workload characterization framework for LLM inference on edge SoCs. Hydra instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema and fuses those records with hardware telemetry, enabling a multi-dimensional characterization of performance, system-resource utilization, and efficiency across prefill and decode phases. Using Hydra, we evaluate three consecutive edge System-on-Chip (SoC) generations (AGX Xavier, AGX Orin, and AGX Thor), 13 instruction-tuned LLMs from seven families, five execution formats, and consider input/output-length sensitivity. The resulting artifact contains roughly 107K per-prompt records and is publicly released with Hydra. Our analysis shows that aggregate latency alone hides key deployment effects: backend structure changes where latency is introduced, quantization reduces memory traffic and energy but does not predict power monotonically, and SoC generation changes how utilization and efficiency should be interpreted. By connecting phase-level timing with system-resource utilization and efficiency metrics, Hydra enables reproducible, phase-aware characterization of edge LLM inference. Hydra's source code and the collected per-prompt trace corpus are available open-source at: https://github.com/amirtaherin/hydra
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。