提出可迁移的延迟预测框架，加速异构设备上大模型部署筛选。
Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices
- 将请求建模为软硬件-模型-提示组合，分阶段预测延迟。
- 在Pixel 8上总延迟R²提升至0.960，解码延迟R²达0.973。
- 轻量校准即可跨设备迁移，适合边缘端快速选型部署。
准确的延迟预测对在异构边缘设备上部署大语言模型至关重要，推理延迟受模型结构、提示行为、运行时后端、硬件利用率、动态电压频率调节（DVFS）和温度变化影响。本文提出一种面向部署的运行时感知延迟预测框架，将每个推理请求表示为硬件-运行时-模型-提示配置，分预填充与解码阶段进行预测，并通过门控模型自适应融合静态特征与动态硬件遥测数据。我们在Pixel移动设备上评估该框架，并在Jetson Nano、Orange Pi 5 Pro和一张RTX 3090级GPU平台上验证了配置流程。在Pixel 8上，完整预测器使总延迟R²从0.953提升至0.960，解码延迟R²从0.957升至0.973；在Pixel 8 Pro上，预填充延迟R²从-1.383提升至0.966。跨设备迁移中，校准使Pixel 8 Pro到Pixel 8的总延迟R²从-0.974提升至0.940，解码延迟R²从-1.085升至0.927。异构分析显示延迟高度依赖设备与运行时：同一SmolLM2模型在Orange Pi 5 Pro上仅达8.42 tokens/s，而在RTX 3090类GPU上可达64.38 tokens/s。结果表明，结合轻量校准的运行时感知预测可显著降低调优成本，支持跨异构边缘平台的延迟敏感型模型部署。
原文摘要 · Abstract (English)
Accurate latency prediction is critical for deploying large language models (LLMs) on heterogeneous edge devices, where inference latency is affected by model architecture, prompt behavior, runtime backend, hardware utilization, dynamic voltage and frequency scaling (DVFS), and thermal variation. This paper presents a runtime-aware latency prediction framework for deployment-oriented LLM selection. The framework represents each inference request as a hardware-runtime-model-prompt configuration, separates inference into prefill and decode phases, and adaptively fuses static descriptors with dynamic hardware telemetry through a gated prediction model. We evaluate the framework using Pixel mobile devices and validate the profiling pipeline on Jetson Nano, Orange Pi 5 Pro, and an RTX 3090-class GPU platform. On Pixel 8, the full predictor improves total-latency R-squared from 0.953 to 0.960 and decode-latency R-squared from 0.957 to 0.973 over a static-only baseline. On Pixel 8 Pro, it improves prefill-latency R-squared from -1.383 to 0.966. For cross-device transfer, calibration improves Pixel 8 Pro to Pixel 8 total-latency R-squared from -0.974 to 0.940 and decode-latency R-squared from -1.085 to 0.927. Heterogeneous profiling further shows that latency is highly device- and runtime-dependent: the same SmolLM2 model family reaches 8.42 tokens/s on Orange Pi 5 Pro but 64.38 tokens/s on an RTX 3090-class GPU. These results demonstrate that runtime-aware prediction with lightweight calibration can reduce profiling cost and support latency-aware LLM deployment across heterogeneous edge platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。