解决内存计算中长文本推理的效率瓶颈，提升性能超10倍。
PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System
- 按令牌划分内存，实现任意批量下的高通道利用率。
- 动态调度指令，重叠数据传输与计算，突破输入输出瓶颈。
- 动态管理缓存内存，消除静态分配浪费，适合真实长文本应用。
长上下文大语言模型的发展带来显著的内存系统挑战。尽管存内计算（PIM）是潜在加速器，但在扩展至长上下文时仍存在严重缺陷：通道利用率低、受制于输入输出瓶颈、静态键值缓存管理导致大量内存浪费。本文提出 PIMphony，一个协同设计的 PIM 协调器，通过三项技术系统性解决上述问题：首先，基于令牌的 PIM 分区（TCP）在任意批大小下保证高通道利用率；其次，动态 PIM 命令调度（DCS）通过重叠数据移动与计算缓解 I/O 瓶颈；最后，动态 PIM 访问控制器（DPA）实现动态内存管理，消除静态内存浪费。基于 MLIR 编译器实现，并在周期精确模拟器上评估，PIMphony 显著提升长上下文大模型推理吞吐量（支持最高 720 亿参数、100 万上下文长度）。实验表明，在纯 PIM 系统上性能提升达 11.3 倍，在 xPU+PIM 系统上提升 8.4 倍，有效推动大模型在真实长文本场景中的高效部署。
原文摘要 · Abstract (English)
The expansion of long-context Large Language Models (LLMs) creates significant memory system challenges. While Processing-in-Memory (PIM) is a promising accelerator, we identify that it suffers from critical inefficiencies when scaled to long contexts: severe channel underutilization, performance-limiting I/O bottlenecks, and massive memory waste from static KV cache management. In this work, we propose PIMphony, a PIM orchestrator that systematically resolves these issues with three co-designed techniques. First, Token-Centric PIM Partitioning (TCP) ensures high channel utilization regardless of batch size. Second, Dynamic PIM Command Scheduling (DCS) mitigates the I/O bottleneck by overlapping data movement and computation. Finally, a Dynamic PIM Access (DPA) controller enables dynamic memory management to eliminate static memory waste. Implemented via an MLIR-based compiler and evaluated on a cycle-accurate simulator, PIMphony significantly improves throughput for long-context LLM inference (up to 72B parameters and 1M context length). Our evaluations show performance boosts of up to 11.3x on PIM-only systems and 8.4x on xPU+PIM systems, enabling more efficient deployment of LLMs in real-world long-context applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。