arXiv:2608.23841cs.ARcs.LG2026-08

为提升CPU上自回归生成的效率,提出软硬件协同设计的新架构与推理引擎。

Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

  • 设计可垂直调度的流水线原生Transformer架构,优化权重访问顺序。
  • 在TinyStories数据集上实现9.00降至4.50 MB/token的带宽减半,仅损失0.24困惑度。
  • 支持异步I/O重叠专家延迟,实测解码速度达5.94 tokens/s,优于现有方案。

在现代CPU上进行单标记自回归生成时,性能受限于内存带宽而非算力:当前CPU算力约1 TFLOP/s,但主存带宽仅为50 GB/s,每生成一个标记需读取所有活跃权重一次。本文主张通过模型架构与推理运行时的协同设计来应对这一瓶颈。提出cflow——一种面向CPU的流式推理引擎,配合一系列流水线原生Transformer架构,其层间依赖图支持垂直、阶段优先的执行调度。cflow将权重按计算消耗顺序分块存储于L2大小的瓷砖中,仅读取每个混合专家层的top-k专家,融合投影操作,并根据模型依赖参数执行延迟感知调度。在五个基于TinyStories训练的架构中,一个(arch2_4_combined)使关键路径权重带宽降低2.00倍(从9.00降至4.50 MB/token),且困惑度仅比最优模型高0.24;瓷砖布局相比行优先基线减少7.29倍的L1数据读取缺失。对于309亿参数的流水线原生MoE模型,cflow在32核Ice Lake服务器上实现5.94 tokens/s的解码速度,优于llama.cpp(4.75)和vLLM CPU后端(1.65)的同类密集模型。将专家延迟窗口实现为磁盘驻留专家层的异步I/O重叠,进一步带来最高1.68倍的净提升,与重叠模型结果相差不足1%。测量验证了八项设计假设中的七项,其中一项被证伪,另一项尚不明确,均已完整报告,附条件说明。

原文摘要 · Abstract (English)

Single-token autoregressive decode on CPUs is bound by memory bandwidth, not arithmetic: a modern CPU sustains roughly 1 TFLOP/s of compute but only about 50 GB/s from main memory, and each generated token must stream every active weight once. This report argues that the most effective response is to co-design the model architecture and the inference runtime together. It presents cflow, a CPU-first streaming engine, alongside a family of pipeline-native transformer architectures whose inter-layer dependency graphs are constructed to permit a vertical, stage-major execution schedule. cflow stores weights as L2-sized tiles in compute-consumption order, reads only the top-k experts of each mixture-of-experts layer, fuses projections, and executes a delay-aware schedule from per-model dependency parameters. Across five architectures trained on TinyStories, one (arch2_4_combined) achieves a 2.00x reduction in critical-path weight bandwidth (9.00 to 4.50 MB/token) within 0.24 perplexity of the best candidate, and the tile layout incurs 7.29x fewer L1-data read misses than a row-major baseline. On a 30.9-billion-parameter pipeline-native MoE, cflow decodes at 5.94 tokens/s (tok/s) on a 32-vCPU Ice Lake server, ahead of llama.cpp (4.75) and the vLLM CPU backend (1.65) on comparably sized dense models. Realizing the expert-delay window as asynchronous I/O overlap on a disk-resident expert tier yields a further net win of up to 1.68x, matching the overlap model within 1%. Measurement refutes one of the eight design claims and leaves a second inconclusive; both are reported in full, with the conditions under which they would hold.

自回归生成推理优化硬件协同

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。