BaseRT让苹果芯片上的大模型推理速度提升1.56倍，性能超越现有方案。
BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal
- 原生基于Metal开发，融合芯片特有优化与统一内存调度。
- 在M3/M4 Pro上实现最高推理吞吐，比llama.cpp快1.56倍。
- 适合追求本地化部署的开发者与注重隐私的边缘计算场景。
我们提出BaseRT，一种原生运行于苹果芯片上的大型语言模型（LLM）推理运行时，实现了该硬件迄今最高的推理吞吐量。现有框架如llama.cpp和MLX-based系统因不匹配Metal执行模型或苹果统一内存拓扑，引入额外开销。通过原生构建于Metal之上，结合芯片级内核融合、统一内存感知优化及自定义分发逻辑，BaseRT恢复了框架方法所丢失的性能。该系统支持所有M系列设备上的八种量化格式（Q2至FP16），涵盖Qwen3、Llama 3.2和Gemma 4等模型家族。在M3和M4 Pro设备上评估Q4和Q8量化模型，BaseRT相较llama.cpp最高提升1.56倍解码吞吐，相较MLX最高提升1.35倍，且在混合专家模型的预填充阶段优势更显著，从子10亿到300亿参数模型均保持最佳性能。这些结果表明苹果芯片作为推理平台的能力强于以往认知，对推动隐私保护、低延迟与云成本压力下的本地化部署具有重要意义。BaseRT已开源：https://github.com/basecompute/baseRT
原文摘要 · Abstract (English)
We present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date. Existing runtimes, including llama.cpp and MLX-based frameworks, incur overhead from abstractions not designed for Metal's execution model or Apple Silicon's unified memory topology. By building natively on Metal with chip-specific kernel fusion, unified memory-aware optimisation, and custom dispatch logic, BaseRT recovers performance that framework-based approaches leave on the table. BaseRT supports a wide range of model families across eight quantisation formats (Q2 to FP16) on all Apple M-series devices. In this paper, we evaluate the Qwen3, Llama 3.2, and Gemma 4 families at Q4 and Q8 quantisation on M3 and M4 Pro devices. BaseRT achieves up to 1.56x higher decode throughput than llama.cpp and up to 1.35x higher than MLX, with substantially larger margins on prefill for mixture-of-experts models, delivering consistent best-in-class throughput from sub-1B to 30B parameter models. These results establish Apple Silicon as a more capable inference platform than previously reported, with direct implications for the emerging edge inference paradigm: as privacy requirements, latency constraints, and cloud cost pressures drive inference toward on-device deployment, performance-optimised local runtimes are a critical enabling layer for this transition. BaseRT is publicly available at https://github.com/basecompute/baseRT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。