实测发现语言模型在苹果芯片上的加速效果取决于表达方式，而非计算内容。
What actually runs: a measurement study of language model placement and decode speed on the Apple Neural Engine
- 通过三种实测方法验证模型部署位置与速度
- 量化编码使模型83%以上运行在神经引擎，速度提升1.8-2.2倍
- 设计应先选编码方式，再分配参数字节数预算
我们研究了语言模型如何被部署到苹果神经引擎（ANE）以及为何能在其上快速运行，通过三项测量回答问题。首先，在64种组合的LLM算子配置中，固定计算内容而改变表达方式，记录每操作的设备支持情况。其次，训练了大小和精度匹配的模型，量化检查点与fp16版本结构字节完全一致，确保每次部署都是真实训练产物。最后，通过读取ANE内存控制器的字节计数器，确认实际运行的内容而非编译器意图。所有结论均经至少两项测量验证。结果表明：模型部署位置是表达方式的属性，而非计算内容；融合的RMSNorm可完全在ANE执行，而其等价分解仅能在CPU运行。权重编码决定加速器使用：一个25.85M参数的卷积密集型fp16模型被完全分配到CPU（计数器显示0字节进入引擎），而相同图在int8或2位下返回约83%驻留率，运行速度提升1.8-2.2倍；较小的22.29M全注意力fp16模型驻留率达98.9%。解码成本为每令牌流过的字节数，恒定约为名义编码宽度的0.77。我们测量的最小最快模型为三值模型，同尺寸下算子分布几乎不变：所有2500万参数三值模型驻留于10.0-10.8 MB，每令牌耗时0.62-0.64毫秒。核心对比：2500万参数半注意力三值模型（10.5 MB，0.63 ms/token）和5000万参数版本（16.8 MB，0.86 ms/token）相比原始卷积密集型fp16设计，体积缩小9.8倍和6.1倍，速度提升3.0倍和2.2倍。基于此，提出设计流程：先选择编码方式，再根据字节预算分配参数量。
原文摘要 · Abstract (English)
We ask what gets a language model onto the Apple Neural Engine (ANE) and what makes it fast there, and we answer with three measurements. We sweep a 64-shape matrix of LLM primitives that varies how a computation is expressed while holding what it computes fixed, recording per-operation device support. We then train matched models across size and precision, with quantized checkpoints byte-identical in structure to their fp16 counterparts, so every deployment measurement is of a real trained artifact. And we read the ANE's memory-controller byte counters during inference, establishing what actually ran rather than what the compiler intended. We support every headline claim with at least two of these three measurement paths. We find that placement is a property of how a computation is expressed, not of what it computes: a fused RMSNorm is fully ANE-eligible while its arithmetically identical decomposition is CPU-only. Weight encoding gates the accelerator: CoreML assigns a 25.85M-parameter conv-heavy fp16 model entirely to the CPU (our counters confirm zero bytes through the engine), while the same graph in int8 or 2-bit returns to ~83% residency and runs 1.8-2.2x faster, and a smaller 22.29M all-attention fp16 model sits at 98.9%. Decode cost is bytes streamed per token, at a constant ~0.77 fraction of nominal encoding width across fp16, int8 and 2-bit. The smallest and fastest models we measured are ternary, and at matched size the operator mix barely moves either axis: every resident 25M ternary model lands within 10.0-10.8 MB and 0.62-0.64 ms/token. The headline pair is half-attention ternary at 25M (10.5 MB, 0.63 ms) and 50M (16.8 MB, 0.86 ms) - 9.8x and 6.1x smaller, 3.0x and 2.2x faster than the conv-heavy fp16 design this work began with. From these measurements we draw a design procedure: choose the encoding first, then spend the byte budget on parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。