MCAP动态评估各层重要性，实现低内存下大模型高效推理。
MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference
- 基于蒙特卡洛采样分析每层激活重要性，指导精度与存储层级分配
- 在NVIDIA T4上实现1.5-1.8倍于llama-cpp Q4_0的解码吞吐
- 无需修改权重即可在极低内存环境下运行大模型，适合边缘部署
将大语言模型部署到异构硬件常受限于内存而非计算能力。我们提出MCAP（蒙特卡洛激活分析），一种加载时的逐层重要性估计器，可在目标设备上实现动态精度与内存放置决策。MCAP生成轻量级逐层信号，驱动精度调度（W4A8 vs. W4A16）与驻留层级（GPU、RAM、SSD），使单组权重在不同内存预算下运行。我们的系统NVE在NVIDIA T4上解码吞吐量比llama-cpp Q4_0高1.5-1.8倍，并支持此前无法实现的低内存运行模式，无需修改权重。
原文摘要 · Abstract (English)
Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than llama-cpp Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。