通过动态剪枝与知识蒸馏，让数学推理大模型更轻更快。
Efficient Mathematical Reasoning Models via Dynamic Pruning and Knowledge Distillation
- 根据权重和熵动态裁剪注意力头，实时降低计算开销。
- 30%剪枝下参数降18.7%，速度提27.5%，准确率仅降0.7%。
- 适合部署在资源受限设备上的数学推理模型优化。
随着深度学习快速发展，大语言模型在数学方程求解等复杂推理任务中表现出色。然而其巨大的计算与存储成本限制了实际应用。本文提出一种轻量级优化方法，融合动态注意力头剪枝与知识蒸馏。该方法结合权重范数与熵值，实时评估多头注意力中各头的重要性，并动态剪除冗余头以降低计算负担。为缓解性能下降，采用知识蒸馏将原模型信息迁移至剪枝后的学生模型，使其保持推理能力。在Math23k与ASDiv-A数据集上的实验验证了该方法的有效性。例如，在Math23k上以30%剪枝率进行剪枝时，参数减少18.7%，推理速度提升27.5%，FLOPs降低19.3%，准确率仅从84.4%降至83.7%。结果表明，该方法在显著提升效率的同时保持了较强的推理性能，为大语言模型在数学推理任务中的高效部署提供了实用方案。
原文摘要 · Abstract (English)
With the rapid development of deep learning, large language models have shown strong capabilities in complex reasoning tasks such as mathematical equation solving. However, their substantial computational and storage costs hinder practical deployment. This paper proposes a lightweight optimization method that integrates dynamic attention head pruning with knowledge distillation. The approach dynamically evaluates the importance of each attention head in the multi-head attention mechanism using a combination of weight norms and entropy, and prunes redundant heads in real time to reduce computational overhead. To mitigate performance degradation, knowledge distillation transfers information from the original model to the pruned student, enabling the smaller model to preserve reasoning ability. Experiments conducted on both Math23k and ASDiv-A verify the effectiveness of the proposed method. For example, on Math23k with a 30% pruning ratio, parameters are reduced by 18.7%, inference speed is improved by 27.5%, FLOPs are reduced by 19.3%, and accuracy drops only 0.7% (from 84.4% to 83.7%). These results demonstrate that the method achieves substantial efficiency gains while maintaining strong reasoning performance, providing a practical solution for efficient deployment of large language models in mathematical reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。