苹果MPS推理出现意外延迟波动，关键在KV缓存与硬件执行模式交互。
Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes
- 发现MPS后端解码延迟非单调，特定配置下延迟飙升21倍。
- 延迟异常源于解码阶段，与内存压力无关，且仅在MPS上出现。
- KV缓存在此类配置下失效，适合关注苹果设备推理优化者阅读。
自回归推理通常被认为随生成长度平滑增长，但本文发现苹果MPS后端在Transformer解码中存在意外的非单调延迟现象。在多个模型（GPT-2、BLOOM、OPT）上，特定解码预算区间内延迟峰值可达21倍，随后在邻近配置迅速恢复。控制实验表明，异常主要出现在解码阶段而非预填充阶段，且无法用内存压力解释，同时在CPU和NVIDIA CUDA后端相同条件下均未出现。进一步分析显示，键值（KV）缓存与异常执行模式强烈耦合：尽管整体仍有益，但在异常配置中加速效果骤降；禁用缓存的解码也表现出残留的非单调行为。这表明MPS上的自回归解码进入离散执行状态，常规粗粒度基准无法捕捉，凸显长上下文推理需硬件感知评估的重要性。
原文摘要 · Abstract (English)
Autoregressive inference is typically assumed to scale predictably with decoding length, with latency increasing smoothly as generated sequence length grows. In this work, we identify unexpected non-monotonic latency behavior in the Apple MPS backend, where latency changes abruptly across nearby decoding configurations during transformer decoding. Using multiple model families (GPT-2, BLOOM, and OPT), we observe latency spikes of up to 21x within specific decoding-budget intervals, followed by recovery at neighboring configurations. Controlled experiments show that these anomalies originate primarily during the decode phase rather than prefill, are not explained by memory pressure alone, and remain absent on CPU and NVIDIA CUDA backends under identical conditions. We further show that key-value (KV) cache interacts strongly with these pathological execution regimes: KV caching remains beneficial overall, but its practical speedup collapses sharply within anomalous configurations, while cache-disabled decoding still exhibits residual non-monotonic behavior. These findings suggest that autoregressive decoding on MPS enters discrete execution regimes that are not captured by coarse-grained benchmarking, highlighting the importance of hardware-aware evaluation for long-context inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。