在苹果M3 Ultra上实现512×512分辨率实时图像生成，突破非CUDA平台优化瓶颈。
Systematic Optimization of Real-Time Diffusion Model Inference on Apple M3 Ultra
- 将蒸馏模型转换为CoreML并配合三线程摄像头管道，实现高效推理。
- 达成22.7帧/秒的实时性能，验证了量化与并行推理在苹果芯片上无效。
- 揭示苹果统一内存架构下的独特优化规律，适合移动端和边缘设备开发者。
尽管基于扩散模型的实时图像生成在NVIDIA GPU上迅速发展，但针对非CUDA平台（如Apple Silicon）的系统性优化研究仍极为有限。本研究在搭载60核GPU、512 GB统一内存的Apple M3 Ultra上，历经10个优化阶段，目标是实现实时相机img2img转换。我们系统评估了CoreML转换、量化、Token Merging、神经引擎利用、紧凑模型探索、帧插值、基于kNN的合成、pix2pix-turbo、光流帧跳过及知识蒸馏等多种技术。最终，通过将专用于蒸馏的SDXS-512模型转为CoreML，并采用三线程摄像头流水线，实现了512×512分辨率下22.7 FPS的实时性能。核心贡献在于揭示：适用于CUDA平台的优化策略在苹果硅统一内存架构下并不适用，其优化路径与NVIDIA GPU有根本差异——量化无加速效果、并行推理无效、神经引擎不适合大模型。本文提供了苹果硅平台扩散模型推理的实用指导。
原文摘要 · Abstract (English)
While real-time image generation using diffusion models has advanced rapidly on NVIDIA GPUs, systematic optimization research on non-CUDA platforms such as Apple Silicon remains extremely limited. In this study, we conducted comprehensive optimization experiments across 10 phases targeting the Apple M3 Ultra (60-core GPU, 512 GB unified memory) with the goal of achieving real-time camera img2img transformation. We explored a wide range of techniques including CoreML conversion, quantization, Token Merging, Neural Engine utilization, compact model exploration, frame interpolation, kNN search-based synthesis, pix2pix-turbo, optical flow frame skipping, and knowledge distillation, quantitatively evaluating the effectiveness of each approach. Ultimately, by combining CoreML conversion of the distillation-specialized model SDXS-512 with a 3-thread camera pipeline, we achieved real-time camera img2img transformation at 22.7 FPS at 512x512 resolution. The primary contribution of this work is the systematic demonstration that optimization insights established for CUDA are not necessarily effective on Apple Silicon's unified memory architecture. We reveal an optimization landscape fundamentally different from that of NVIDIA GPUs -- including the absence of speedup from quantization, the ineffectiveness of parallel inference, and the unsuitability of the Neural Engine for large-scale models -- and provide practical guidelines for diffusion model inference on Apple Silicon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。