用模块感知优化提升大模型效率，降低能耗同时保持性能。
MAR: Efficient Large Language Models via Module-aware Architecture Refinement
- 引入状态空间模型与激活稀疏化，线性化序列处理并减少计算开销。
- 在有限资源下实现与密集模型相当的性能，推理能耗显著下降。
- 适合追求高效部署的大模型应用，如边缘设备或实时系统。
大型语言模型（LLMs）在多个领域表现优异，但因注意力机制的二次复杂度和密集前馈网络（FFN）运算导致高能耗。为此，我们提出模块感知架构优化（MAR），一种两阶段框架：采用状态空间模型（SSMs）实现线性时间序列建模，并通过激活稀疏化降低FFN成本。此外，为缓解脉冲神经网络（SNNs）与SSMs集成中的信息密度低和时序不匹配问题，设计了自适应三值多步神经元（ATMN）和脉冲感知双向蒸馏策略（SBDS）。大量实验表明，MAR在资源受限条件下有效恢复其密集版本的性能，同时大幅降低推理能耗。且其性能优于同等或更大规模的高效模型，展现出构建高效实用大模型的巨大潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel across diverse domains but suffer from high energy costs due to quadratic attention and dense Feed-Forward Network (FFN) operations. To address these issues, we propose Module-aware Architecture Refinement (MAR), a two-stage framework that integrates State Space Models (SSMs) for linear-time sequence modeling and applies activation sparsification to reduce FFN costs. In addition, to mitigate low information density and temporal mismatch in integrating Spiking Neural Networks (SNNs) with SSMs, we design the Adaptive Ternary Multi-step Neuron (ATMN) and the Spike-aware Bidirectional Distillation Strategy (SBDS). Extensive experiments demonstrate that MAR effectively restores the performance of its dense counterpart under constrained resources while substantially reducing inference energy consumption. Furthermore, it outperforms efficient models of comparable or even larger scale, underscoring its potential for building efficient and practical LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。