在苹果芯片上实现大模型高效推理，支持文本与多模态任务
Native LLM and MLLM Inference at Scale on Apple Silicon
- 基于MLX原生构建，支持连续批处理和内容感知缓存
- 文本模型吞吐提升21%至87%，重复图像查询加速28倍
- 适合在Mac上部署大模型的开发者和研究者使用
随着苹果芯片在机器学习开发中的普及，对利用其统一内存架构的高效推理方案需求日益增长。现有工具或缺乏原生优化（如PyTorch MPS），或仅聚焦文本模型，导致多模态任务支持不足。本文提出vllm-mlx，一个基于MLX的原生框架，用于苹果芯片上的大语言模型（LLM）与多模态大模型（MLLM）高效推理。对于文本模型，在Qwen3-0.6B至Nemotron-30B范围内，相比llama-cpp实现21%至87%的吞吐提升，连续批处理在16个并发请求下可实现4.3倍聚合吞吐。针对多模态模型，引入基于内容哈希的前缀缓存机制，无论输入格式如何，均可识别相同图像并避免重复视觉编码。在Apple M4 Max上评估显示，文本模型最高达525 tokens/秒，重复图像查询延迟从21.7秒降至1秒以下，加速28倍；视频分析最多64帧时，缓存加速达24.7倍。代码已开源，支持消费级苹果硬件上的高效推理。
原文摘要 · Abstract (English)
The growing adoption of Apple Silicon for machine learning development has created demand for efficient inference solutions that leverage its unique unified memory architecture. However, existing tools either lack native optimization (PyTorch MPS) or focus solely on text models, leaving multimodal workloads underserved. We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built natively on MLX. For text models, we achieve 21\% to 87\% higher throughput than llama-cpp across models ranging from Qwen3-0.6B to Nemotron-30B, while providing continuous batching that scales to 4.3x aggregate throughput at 16 concurrent requests. For multimodal models, we introduce content-based prefix caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format. Our evaluation on Apple M4 Max demonstrates throughput of up to 525 tokens per second on text models and 28x speedup on repeated image queries, reducing multimodal latency from 21.7 seconds to under 1 second. Video analysis with up to 64 frames achieves 24.7x cache speedup. We release our implementation as open source to support efficient inference on consumer Apple hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。