DAOP动态分配专家，提升MoE模型在设备端的推理效率。
DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference
- 根据序列激活模式动态分配专家到CPU或GPU
- 预计算预测专家，减少数据传输延迟8.20倍
- 适合资源受限设备上的高效MoE部署
混合专家（MoE）模型虽在多种机器学习任务中表现优异，但在内存受限设备上部署面临挑战。尽管GPU推理速度快，但其内存远小于CPU，无法同时存储所有专家，导致频繁且代价高昂的数据迁移，常抵消GPU优势。为此，我们提出DAOP，一种面向设备端的MoE推理引擎，优化GPU-CPU并行执行。DAOP基于每条输入的激活模式动态分配专家至CPU或GPU，并在CPU上选择性预计算可能激活的专家，以最小化传输延迟。该方法在不同专家缓存比例下均实现高效资源利用，并通过新型渐进退化机制保持模型精度。在多个数据集上的综合评估表明，相比传统专家缓存与预取方法，DAOP最高提速8.20倍；相比卸载技术，提速1.35倍，且精度不受损。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models, though highly effective for various machine learning tasks, face significant deployment challenges on memory-constrained devices. While GPUs offer fast inference, their limited memory compared to CPUs means not all experts can be stored on the GPU simultaneously, necessitating frequent, costly data transfers from CPU memory, often negating GPU speed advantages. To address this, we present DAOP, an on-device MoE inference engine to optimize parallel GPU-CPU execution. DAOP dynamically allocates experts between CPU and GPU based on per-sequence activation patterns, and selectively pre-calculates predicted experts on CPUs to minimize transfer latency. This approach enables efficient resource utilization across various expert cache ratios while maintaining model accuracy through a novel graceful degradation mechanism. Comprehensive evaluations across various datasets show that DAOP outperforms traditional expert caching and prefetching methods by up to 8.20x and offloading techniques by 1.35x while maintaining accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。