测试生成式AI在手机电脑上的性能表现，发现资源争抢导致延迟飙升。
ConsumerBench: Benchmarking Generative AI Applications on End-User Devices
- 模拟多应用并发运行，真实还原设备端使用场景
- 发现静态服务器配置导致性能下降，资源调度不公
- 适合模型开发者和系统设计者优化移动端生成式AI
生成式AI应用正从云端转向终端设备，带来资源管理、系统效率与用户体验的新挑战。本文提出ConsumerBench，一个全面的基准测试框架，用于评估生成式AI模型在终端设备上的系统效率与响应时间。不同于以往假设独占专用GPU的基准，ConsumerBench模拟真实环境中多个应用在资源受限设备上并发运行的场景。该框架支持可定制工作流，模拟需多应用协同的复杂任务，并采集应用级指标（如延迟、服务等级目标SLO达成率）与系统级指标（如CPU/GPU利用率、内存带宽）。大量实验揭示了资源共享低效、贪婪分配策略下的不公平调度以及静态模型服务器配置的性能缺陷。论文还为模型开发者和系统设计者提供实用建议，强调针对消费级GPU架构定制内核及实现SLO感知调度策略的价值。
原文摘要 · Abstract (English)
The recent shift in Generative AI (GenAI) applications from cloud-only environments to end-user devices introduces new challenges in resource management, system efficiency, and user experience. This paper presents ConsumerBench, a comprehensive benchmarking framework designed to evaluate the system efficiency and response time of GenAI models running on end-user devices. Unlike existing benchmarks that assume exclusive model access on dedicated GPUs, ConsumerBench simulates realistic multi-application scenarios executing concurrently on constrained hardware. Furthermore, ConsumerBench supports customizable workflows that simulate complex tasks requiring coordination among multiple applications. ConsumerBench captures both application-level metrics, including latency and Service Level Objective (SLO) attainment, and system-level metrics like CPU/GPU utilization and memory bandwidth. Through extensive experiments, ConsumerBench reveals inefficiencies in resource sharing, unfair scheduling under greedy allocation, and performance pitfalls of static model server configurations. The paper also provides practical insights for model developers and system designers, highlighting the benefits of custom kernels tailored to consumer-grade GPU architectures and the value of implementing SLO-aware scheduling strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。