通过评估权重重要性，实现大模型剪枝的精准控制。
Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability
- 动态追踪训练中权重变化，系统评估每项参数重要性。
- 适度剪枝可提升效率并降低损失，过度剪枝则严重退化性能。
- 适合关注大模型节能与可持续发展的研究者和工程师。
以ChatGPT为代表的大语言模型（LLM）在自然语言处理领域取得突破，但其训练所需巨大算力带来显著环境影响，包括高碳排放、能耗与水资源消耗。本文提出一种新型大模型剪枝方法，聚焦于训练过程中个体权重重要性的系统评估。通过监测参数随时间演变，该方法可在不损害性能的前提下有效压缩模型规模。对简化版LLM及大型多模态模型的广泛实验表明，适度剪枝能提升效率并降低损失，而过度剪枝将导致性能急剧下降。结果凸显优化模型对实现可持续人工智能发展的重要性，平衡技术进步与环境责任。
原文摘要 · Abstract (English)
The exponential growth of large language models (LLMs) like ChatGPT has revolutionized artificial intelligence, offering unprecedented capabilities in natural language processing. However, the extensive computational resources required for training these models have significant environmental implications, including high carbon emissions, energy consumption, and water usage. This research presents a novel approach to LLM pruning, focusing on the systematic evaluation of individual weight importance throughout the training process. By monitoring parameter evolution over time, we propose a method that effectively reduces model size without compromising performance. Extensive experiments with both a scaled-down LLM and a large multimodal model reveal that moderate pruning enhances efficiency and reduces loss, while excessive pruning drastically deteriorates model performance. These findings highlight the critical need for optimized AI models to ensure sustainable development, balancing technological advancement with environmental responsibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。