系统评估大模型效率技术，揭示不同方法的权衡与适用场景。
EfficientLLM: Efficiency in Large Language Models
- 构建首个大规模实证基准，覆盖架构、微调、推理三类优化方法。
- 发现40%显存增加可换得更低计算开销，量化能降3.9倍能耗但精度略降。
- 技术效果随任务和模型规模变化，且在视觉模型上可迁移应用。
大语言模型推动了显著进展，但其参数量和上下文窗口的增长带来了高昂的计算、能源与资金成本。我们提出 EfficientLLM，首个全面的实证研究，系统评估大模型效率技术。在生产级集群（48xGH200, 8xH200 GPU）上，研究涵盖三大维度：（1）架构预训练（高效注意力：MQA、GQA、MLA、NSA；稀疏专家混合MoE），（2）微调（参数高效方法：LoRA、RSLoRA、DoRA），（3）推理（量化：int4、float16）。定义六项细粒度指标（内存利用率、计算利用率、延迟、吞吐、能耗、压缩率）以捕捉硬件饱和度、延迟-吞吐平衡与碳排放。评估超过100个模型-技术组合（0.5B–72B参数），得出三项核心洞察：（i）效率涉及可量化的权衡：无单一方法普适最优；如MoE降低FLOPs并提升准确率，但显存增加40%；int4量化使内存/能耗下降至3.9倍，精度损失3–5%。（ii）最优方案依赖任务与规模：MQA在资源受限设备中表现最佳，MLA在质量敏感任务中实现最低困惑度，而RSLoRA仅在超14B参数时优于LoRA。（iii）技术具备跨模态通用性：扩展至大视觉模型（Stable Diffusion 3.5、Wan 2.1）和视觉语言模型（Qwen2.5-VL），验证有效性。通过开源数据集、评估流程与排行榜，EfficientLLM为下一代基础模型的效率-性能权衡提供关键指导。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have driven significant progress, yet their growing parameter counts and context windows incur prohibitive compute, energy, and monetary costs. We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale. Conducted on a production-class cluster (48xGH200, 8xH200 GPUs), our study systematically explores three key axes: (1) architecture pretraining (efficient attention variants: MQA, GQA, MLA, NSA; sparse Mixture-of-Experts (MoE)), (2) fine-tuning (parameter-efficient methods: LoRA, RSLoRA, DoRA), and (3) inference (quantization methods: int4, float16). We define six fine-grained metrics (Memory Utilization, Compute Utilization, Latency, Throughput, Energy Consumption, Compression Rate) to capture hardware saturation, latency-throughput balance, and carbon cost. Evaluating over 100 model-technique pairs (0.5B-72B parameters), we derive three core insights: (i) Efficiency involves quantifiable trade-offs: no single method is universally optimal; e.g., MoE reduces FLOPs and improves accuracy but increases VRAM by 40%, while int4 quantization cuts memory/energy by up to 3.9x at a 3-5% accuracy drop. (ii) Optima are task- and scale-dependent: MQA offers optimal memory-latency trade-offs for constrained devices, MLA achieves lowest perplexity for quality-critical tasks, and RSLoRA surpasses LoRA efficiency only beyond 14B parameters. (iii) Techniques generalize across modalities: we extend evaluations to Large Vision Models (Stable Diffusion 3.5, Wan 2.1) and Vision-Language Models (Qwen2.5-VL), confirming effective transferability. By open-sourcing datasets, evaluation pipelines, and leaderboards, EfficientLLM provides essential guidance for researchers and engineers navigating the efficiency-performance landscape of next-generation foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。