研究边缘GPU上大模型推理的性能平衡,给出实用部署方案。
EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs
- 系统测试不同模型规模与架构的延迟-准确率权衡
- 验证减少推理令牌长度的技术在保持性能下的有效性
- 适合需要低延迟、高效率的边缘智能应用开发者
边缘智能正被新兴自主系统(如机器人)广泛需求。相较于云端方案,边缘部署可实现隐私保护、网络弱环境下稳定运行,并具备显著能效与成本优势。然而,在边缘GPU上部署用于推理的大语言模型(LLMs)面临严格延迟约束和有限计算资源的挑战。开发者需在推理架构选择、模型大小、令牌预算分配及测试时缩放策略间权衡,以满足目标延迟并优化准确率。但现有对这些变量最优组合的指导仍不足。本文提出EdgeReasoning,系统性地刻画了推理型LLM在边缘GPU上的部署特性。我们量化了多种LLM架构与模型规模下的延迟-准确率权衡关系;评估了基于提示和模型微调的技术在压缩推理令牌长度的同时维持性能的效果;进一步分析了不同并行度的测试时缩放方法在严格延迟预算下的准确率上限。通过上述分析,本工作绘制出可达的准确率-延迟帕累托前沿,为推理型LLM在边缘端的最优部署提供系统性指导。
原文摘要 · Abstract (English)
Edge intelligence paradigm is increasingly demanded by the emerging autonomous systems, such as robotics. Beyond ensuring privacy-preserving operation and resilience in connectivity-limited environments, edge deployment offers significant energy and cost advantages over cloud-based solutions. However, deploying large language models (LLMs) for reasoning tasks on edge GPUs faces critical challenges from strict latency constraints and limited computational resources. To navigate these constraints, developers must balance multiple design factors - choosing reasoning versus non-reasoning architectures, selecting appropriate model sizes, allocating token budgets, and applying test-time scaling strategies - to meet target latency and optimize accuracy. Yet guidance on optimal combinations of these variables remains scarce. In this work, we present EdgeReasoning, a comprehensive study characterizing the deployment of reasoning LLMs on edge GPUs. We systematically quantify latency-accuracy tradeoffs across various LLM architectures and model sizes. We systematically evaluate prompt-based and model-tuning-based techniques for reducing reasoning token length while maintaining performance quality. We further profile test-time scaling methods with varying degrees of parallelism to maximize accuracy under strict latency budgets. Through these analyses, EdgeReasoning maps the Pareto frontier of achievable accuracy-latency configurations, offering systematic guidance for optimal edge deployment of reasoning LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。