让苹果神经引擎支持大模型训练与推理，突破系统限制实现高效本地运行。
Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference
- 直接调用私有API绕过CoreML，构建端到端可训练系统
- 通过权重文件热更新，训练速度提升3.8倍，每步仅需494毫秒
- 支持110M参数模型稳定训练，适合开发者在Mac上部署轻量级LLM
超过二十亿台苹果设备搭载了神经网络处理器（NPU）——苹果神经引擎（ANE），但该加速器在大语言模型任务中仍基本未被使用。Apple的CoreML框架采用不透明抽象，无法直接编程且不支持设备端训练。本文提出Orion，据我们所知首个开源的端到端系统，结合直接ANE执行、编译器流水线与稳定多步训练及断点续训功能，完全绕过CoreML，通过Apple私有接口 _ANEClient 与 _ANECompiler 实现。基于maderix的前期研究，我们扩展了对ANE约束的认知，整理出20项限制，包含14项此前未公开的约束。Orion包含一个五阶段优化编译器，将图式IR转换为ANE原生MIL，并配备管理IOSurface零拷贝张量输入输出、程序缓存与增量编译的运行时。由于ANE在编译时固化权重，传统训练需每步全重编译（约4.2秒）。我们发现可通过卸载、修改权重文件并重新加载，跳过ANECCompile()，将重编译时间从4,200毫秒降至494毫秒（8.5倍加速），训练速度提升3.8倍。在M4 Max上，Orion实现GPT-2 124M推理超170 tokens/s，成功在TinyStories数据集上稳定训练110M参数Transformer达1,000步，耗时22分钟，无任何NaN发生。此外，提出LoRA适配器作为输入，通过IOSurface实现适配器热替换，无需重新编译。
原文摘要 · Abstract (English)
Over two billion Apple devices ship with a Neural Processing Unit (NPU) - the Apple Neural Engine (ANE) - yet this accelerator remains largely unused for large language model workloads. CoreML, Apple's public ML framework, imposes opaque abstractions that prevent direct ANE programming and do not support on-device training. We present Orion, to our knowledge the first open end-to-end system that combines direct ANE execution, a compiler pipeline, and stable multi-step training with checkpoint resume in a single native runtime, bypassing CoreML entirely via Apple's private _ANEClient and _ANECompiler APIs. Building on prior characterization work by maderix, we extend public knowledge of ANE constraints to a catalog of 20 restrictions on MIL IR programs, memory layout, compilation limits, and numerical behavior, including 14 previously undocumented constraints discovered during Orion development. Orion includes a compiler that lowers a graph IR through five optimization passes to ANE-native MIL and a runtime that manages IOSurface-backed zero-copy tensor I/O, program caching, and delta compilation for weight updates. Because the ANE bakes weights at compile time, naive training normally requires full recompilation per step (~4.2 s). We show that compiled programs can instead be updated by unloading, patching weight files, and reloading, bypassing ANECCompile() and reducing recompilation from 4,200 ms to 494 ms per step (8.5x), yielding a 3.8x training speedup. On an M4 Max, Orion achieves 170+ tokens/s for GPT-2 124M inference and demonstrates stable training of a 110M-parameter transformer on TinyStories for 1,000 steps in 22 minutes with zero NaN occurrences. We also present LoRA adapter-as-input, enabling hot-swap of adapters via IOSurface inputs without recompilation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。