arXiv:2604.27115cs.CL2026-04

发现任务特定神经元,可高效压缩模型且不影响性能

Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models

论文配图:Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models
图 1 · 摘自论文原文
  • 用激活度筛选关键神经元,实现精准剪枝
  • 删掉10%核心神经元导致性能彻底崩溃
  • 剪枝后微调能有效恢复模型能力,适合部署优化

神经元剪枝被广泛用于降低大语言模型的计算开销和参数量,但任务特定模型中的神经元是否均匀贡献仍不明确。本文通过对数学推理与代码生成专用语言模型进行系统性剪枝实验,提出基于激活度的选择性度量方法,识别对目标任务贡献低的神经元并予以剪除,同时保持目标任务准确率。相比随机剪枝,选择性剪枝表现更优。反向剪枝实验显示,仅移除约10%高度任务特异的神经元即引发性能完全崩溃,表明存在任务特定神经元,且关键信息集中于网络小部分。而剪除约30%-35%非关键神经元虽降低准确率,仍保留显著性能。在1.5B和7B模型上均观察到约15%-20%为鲁棒性阈值,超过则准确率骤降、生成失败增多。微调可显著恢复各剪枝水平下的性能,尤其对激进剪枝模型效果明显。研究为任务特定语言模型中神经元专业化提供了实证支持,并揭示了剪枝鲁棒性、模型冗余与剪枝后恢复机制。

原文摘要 · Abstract (English)

Neuron pruning is widely used to reduce the computational cost and parameter footprint of large language models, yet it remains unclear whether neurons in task-specific models contribute uniformly to task performance. In this work, we provide empirical evidence for the existence and importance of task-specific neurons through a systematic pruning study on language models specialized for mathematical reasoning and code generation. We introduce an activation-based selectivity metric to identify neurons with low contribution to the target task and prune them while preserving target-task accuracy, and compare selective pruning with random pruning. Selective pruning consistently outperforms random pruning, indicating that activation-based selectivity provides a systematic advantage over random pruning. Reverse pruning experiments further show that removing a small subset of highly task-specific neurons (~10%) causes complete performance collapse, suggesting that there exist task specific neurons and critical task information is concentrated in a small portion of the network. In contrast, selective pruning of less critical neurons (~30% - ~35%) reduces accuracy but still preserves significant performance. We also observed consistent reductions in parameters and runtime VRAM usage, along with improved inference throughput as pruning increases. Experiments on both 1.5B and 7B models reveal a robustness threshold around 15-20% pruning, beyond which accuracy loss and generation failures increase sharply. Fine-tuning substantially recovers performance across pruning levels, particularly for aggressively pruned models. These findings provide empirical evidence of neuron specialization in task-specific language models and offer insights into pruning robustness, model redundancy, and post-pruning recoverability.

神经元剪枝模型压缩任务特定微调恢复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。