通过剪枝特定神经元提升大模型泛化能力。
Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models
- 用积分梯度识别导致过拟合的神经元。
- 在多选题测试中性能超越非剪枝方法。
- 适合需要强泛化的模型优化场景。
大型语言模型（LLMs）常针对特定数据集发展出专用机制，如依赖领域特异性相关性，产生高置信度预测但缺乏可迁移推理能力。这类机制在新任务或分布下通常导致性能下降。本文提出一种微调方法，通过集成梯度量化神经元对高置信度预测的影响，识别并剪枝与数据集特异性机制相关的神经元，迫使模型依赖更具泛化性的表示。在多个多项选择基准上评估，基于剪枝的微调显著优于先前（非剪枝）适配方法。
原文摘要 · Abstract (English)
Large language models (LLMs) often develop learned mechanisms specialized to specific datasets, such as reliance on domain-specific correlations, which yield high-confidence predictions without generalizable reasoning. While beneficial in one setting, these dataset-specific mechanisms typically degrade performance when models encounter novel tasks or distributions. In this work, we introduce a fine-tuning approach designed to enhance generalization by identifying and pruning neurons associated with dataset-specific mechanisms in transformer-based LLMs. Our method employs Integrated Gradients to quantify each neuron's influence on high-confidence predictions, pinpointing those that disproportionately contribute to dataset-specific performance without supporting robust, transferable reasoning. Selectively pruning these neurons compels the model to depend on generalizable representations. Evaluated across multiple-choice benchmarks, our pruning-based fine-tuning significantly enhances performance, surpassing prior (non-pruning) adaptation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。