FluidML让模型在边缘设备上更快更省内存地运行。
FluidML: Fast and Memory Efficient Inference Optimization
- 动态调整模型执行流程，智能管理内存使用
- 推理延迟降低最高25.38%，峰值内存减少41.47%
- 轻量开源框架，适配多种边缘设备应用
部署于边缘设备的机器学习模型推动了人形机器人、AR眼镜和自动驾驶等新应用的发展。然而，这些设备的计算资源难以跟上模型参数数量的增长。随着模型规模增大和结构复杂化，新型架构对推理优化提出了新挑战。本文提出FluidML，一个通用的运行时内存管理与优化框架，可灵活重构模型执行蓝图，实现更快、更高效的推理。跨多平台测试表明，FluidML在主流语言模型上可将端到端推理延迟最高降低25.38%，峰值内存使用量最多减少41.47%，优于现有先进方法。该框架代码量约3万行，面向通用场景设计，将作为开源推理优化框架发布。
原文摘要 · Abstract (English)
Machine learning models deployed on edge devices have enabled numerous exciting new applications, such as humanoid robots, AR glasses, and autonomous vehicles. However, the computing resources available on these edge devices are not catching up with the ever-growing number of parameters in these models. As the models become bigger and more complicated, the novel yet sophisticated structure challenges the inference runtime optimization. We present FluidML, a generic runtime memory management and optimization framework that can flexibly transform the model execution blueprint to achieve faster and more memory-efficient inference. Evaluations across different platforms show that FluidML can consistently reduce the end-to-end inference latency by up to 25.38% for popular language models and reduce peak memory usage by up to 41.47%, compared to state-of-the-art approaches. FluidML is of ~30K line of codes, built for general-purpose usage, and will be released as an open-source inference runtime optimization framework to the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。