苹果芯片上实现CPU+GPU协同加速推理，突破了此前精度与性能的矛盾。
FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon
- 通过感知竞争的分层行拆分策略，实现变压器预填充阶段的跨单元协同计算。
- 在五款芯片上实测，预填充速度提升1.15至1.38倍，首令牌生成更快1.18至1.25倍。
- 适用于推理场景中的预填充阶段，不适用于解码或训练，且对内存压力敏感。
苹果自研芯片采用统一内存架构，共享CPU、GPU和神经引擎。本文探讨是否可通过跨单元拆分单个算子来加速Transformer推理。此前尝试均因精度问题失败。研究发现：MLX的懒加载调度器在跨流操作中会强制串行化，导致行拆分虽在输入就绪时快1.38倍，但在懒加载图中反而比纯GPU慢0.66倍；而通过显式材料化边界恢复并发性后，加速比达1.34倍。本系统基于此机制，在每层引入感知竞争的行拆分策略，用于变压器预填充。在三代苹果芯片、五个型号上验证，社区复现结果一致：针对类Llama解码块的预填充加速1.15–1.38倍，32层深度下无退化；真实Qwen2.5-7B模型通过原生MLX-LM服务，首令牌生成提速1.18–1.25倍，输出完全一致，解码吞吐不变。同时明确边界：解码无法受益，受共享带宽限制；精度匹配训练在所有芯片上退化0.86–0.97倍；神经引擎层级调度开销过大；运行时无退化门控在内存压力下失效，探查替代模式会挤出活跃工作集。代码、原始数据与生成日志已公开。
原文摘要 · Abstract (English)
Apple-Silicon SoCs share CPU, GPU, and Neural Engine over one unified memory system, raising the question of whether transformer inference can be accelerated by splitting single operators across units. Prior attempts, including our own, failed or produced precision-confounded wins. We identify the cause: MLX's lazy-graph scheduler \emph{serializes} cross-stream work whenever a CPU-stream operation consumes an unmaterialized GPU result inside one evaluation graph, so a row-split matmul that runs \x{1.38} faster with materialized inputs runs \x{0.66} slower than GPU-only inside a lazy graph; an eager materialization boundary restores concurrency (\x{1.34}). \sys{} implements a per-layer, contention-aware CPU+GPU row split for transformer prefill built on this fix. Evaluated across five chips and three Apple-Silicon generations, community-replicated, the split accelerates Llama-shaped decoder-block prefill by \x{1.15}--\x{1.38}, unchanged at full 32-block depth, and reaches \x{1.18}--\x{1.25} faster time-to-first-token on a real Qwen2.5-7B checkpoint served through stock MLX-LM, with token-identical outputs and unchanged decode throughput. We characterize the boundaries equally carefully: decode cannot benefit, bound by shared bandwidth co-execution does not add; precision-matched training loses \x{0.86}--\x{0.97} on all five chips; ANE dispatch overhead excludes it at layer granularity; and a no-regression runtime gate becomes self-defeating under memory pressure, where probing an alternative mode evicts the active mode's working set. Code, raw results, and generation transcripts are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。