对比5个苹果芯片本地大模型推理框架，实测性能与适用场景。
Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS
- 在M2 Ultra上测试五款框架，覆盖生成速度、长文本处理等指标。
- MLX吞吐最高，MLC-LLM首字延迟低，llama.cpp适合轻量单流任务。
- 适合隐私敏感、需本地部署的开发者，尤其关注长上下文和实时性。
我们对五种苹果芯片上的本地大语言模型（LLM）运行时——MLX、MLC-LLM、llama.cpp、Ollama 和 PyTorch MPS——进行了系统性、实证评估。实验在配备M2 Ultra处理器和192 GB统一内存的Mac Studio上进行，使用Qwen-2.5模型家族，在从数百到10万词元的提示长度下，测量了首次词元时间（TTFT）、稳定生成吞吐率、延迟分位数、长上下文行为（键值缓存与提示缓存）、量化支持、流式传输性能、批处理与并发行为及部署复杂度。结果显示：MLX在持续生成吞吐上表现最优；MLC-LLM在中等提示长度下保持更低的TTFT，且开箱即用功能更强；llama.cpp在轻量单流场景下效率高；Ollama注重开发体验但吞吐和首字延迟落后；PyTorch MPS因内存限制在大模型与长上下文中表现受限。所有框架均实现全设备端运行，无遥测，保障强隐私。我们开源了脚本、日志和图表以复现结果。分析揭示了苹果生态下LLM部署的设计权衡，并为交互式与长上下文任务提供数据驱动建议。尽管当前性能仍不及NVIDIA GPU系统（如vLLM），但苹果芯片推理框架正迅速成熟为可生产使用的私有化本地推理方案。
原文摘要 · Abstract (English)
We present a systematic, empirical evaluation of five local large language model (LLM) runtimes on Apple Silicon: MLX, MLC-LLM, llama.cpp, Ollama, and PyTorch MPS. Experiments were conducted on a Mac Studio equipped with an M2 Ultra processor and 192 GB of unified memory. Using the Qwen-2.5 model family across prompts ranging from a few hundred to 100,000 tokens, we measure time-to-first-token (TTFT), steady-state throughput, latency percentiles, long-context behavior (key-value and prompt caching), quantization support, streaming performance, batching and concurrency behavior, and deployment complexity. Under our settings, MLX achieves the highest sustained generation throughput, while MLC-LLM delivers consistently lower TTFT for moderate prompt sizes and offers stronger out-of-the-box inference features. llama.cpp is highly efficient for lightweight single-stream use, Ollama emphasizes developer ergonomics but lags in throughput and TTFT, and PyTorch MPS remains limited by memory constraints on large models and long contexts. All frameworks execute fully on-device with no telemetry, ensuring strong privacy guarantees. We release scripts, logs, and plots to reproduce all results. Our analysis clarifies the design trade-offs in Apple-centric LLM deployments and provides evidence-based recommendations for interactive and long-context processing. Although Apple Silicon inference frameworks still trail NVIDIA GPU-based systems such as vLLM in absolute performance, they are rapidly maturing into viable, production-grade solutions for private, on-device LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。