针对多核CPU推理瓶颈，提出轻量级架构提升吞吐46%。
ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs
- 从零设计架构，优化内存管理与线程调度
- 跨NUMA访问开销降低，推理吞吐最高提升46%
- 兼容任意CPU设备，适合服务器和网络场景
尽管现有大型语言模型（LLM）在CPU上的推理框架已较成熟，但仍未能充分挖掘多核CPU平台的计算潜力。多核CPU广泛部署于网络服务器与高端网络设备中，通常由多个NUMA节点组成，每个节点包含一组核心与本地内存。当前框架普遍忽视跨NUMA内存访问带来的显著开销，限制了推理的可扩展性与智能能力。为此，我们构建了ArcLight——一个专为多核CPU从零设计的轻量级LLM推理架构。该架构融合高效的内存管理与线程调度机制，并引入细粒度控制的张量并行策略，以缓解跨节点内存访问瓶颈。实验表明，ArcLight显著突破主流框架性能上限，在多种测试场景下实现最高达46%的推理吞吐提升。此外，系统保持对任意CPU设备的兼容性。代码已公开于https://github.com/OpenBMB/ArcLight。
原文摘要 · Abstract (English)
Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。