针对AMD GPU优化大模型推理，发现架构适配是关键。
Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
- 按模型架构定制推理参数，如MLA需块大小为1
- 405B模型在高并发下吞吐达1.6万词元/秒，视觉任务最高超6倍
- 适合部署大模型的工程师和追求性能极致的团队
我们对AMD Instinct MI325X GPU上的生产级大模型推理进行了跨架构评估，在8卡集群（共2TB HBM3e）上测试了四个从2350亿到万亿参数的模型，涵盖MoE+MLA、Dense+GQA和MoE+GQA三类架构。结果表明：架构感知优化至关重要——MLA模型需设块大小为1且不可启用KV缓存卸载，而GQA模型则两者均受益。使用AMD AITER运行时可提升MLA推理吞吐，但对不兼容注意力头配置的架构需选择性禁用。对Llama-3.1-405B的控制实验显示，高并发下仅获3-5%吞吐增益，但测量波动达2-16倍，证实AITER加速主要针对MoE/MLA内核。文本任务中，Llama-405B与DeepSeek V3.2峰值吞吐相近（分别为15,944和15,343 tok/s），尽管活跃参数量差一个数量级。视觉任务中，Qwen3-VL-235B达到47,873 tok/s，是Kimi-K2.5（7,327 tok/s）的6.5倍。活跃参数数与吞吐相关，但受量化、AITER加速和张量并行影响。所有模型在短序列下约500并发时出现吞吐饱和，长序列为100-200并发，均为内存带宽瓶颈。全部模型在1000并发下保持100%成功率，处理1890万词元、17,406请求无失败。
原文摘要 · Abstract (English)
We present a cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs, benchmarking four models spanning 235B to 1 trillion parameters across three architectural families (MoE+MLA, Dense+GQA, MoE+GQA) on an 8-GPU cluster with 2TB aggregate HBM3e using vLLM v0.14.1. Our results demonstrate that architecture-aware optimization is essential: MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both. The AMD AITER runtime is required for competitive MLA inference throughput and must be selectively disabled for architectures with incompatible attention head configurations. A controlled AITER ablation on Llama-3.1-405B (n=5 per condition) reveals a modest 3-5% throughput benefit at high concurrency but 2-16x higher measurement variability, confirming that AITER's large speedups target MoE/MLA kernels specifically. Under text-only workloads, Llama-405B and DeepSeek V3.2 achieve comparable peak throughput (15,944 and 15,343 tok/s) despite an order-of-magnitude difference in active parameters. Under vision workloads, Qwen3-VL-235B reaches 47,873 tok/s, 6.5x higher than Kimi-K2.5 (7,327 tok/s). Active parameter count per token is associated with inference throughput, though confounded by differences in quantization, AITER acceleration, and tensor parallelism. All four models exhibit a common throughput saturation point consistent with a memory-bandwidth bottleneck (~500 concurrent for short sequences, ~100-200 for longer sequences). All models maintain 100% HTTP-level success rates through 1,000 concurrent users, processing 18.9 million tokens across 17,406 requests without failures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。