arXiv:2505.23811cs.CLcs.AI2025-05NeurIPS被引 15

用影响函数评估大模型各层质量,让不同任务下层的作用更精准。

LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions

  • 基于影响函数分析每层对验证损失的敏感度,量化训练质量。
  • 同一模型在不同任务中显示层重要性差异,揭示层的专用性。
  • 可指导LoRA-MoE专家分配和模型剪枝,提升任务性能。

预训练大语言模型在多种任务上表现优异,但各层训练质量存在显著差异,影响下游性能。现有方法多依赖模型中心的启发式策略(如谱统计、异常检测或均分),忽视了数据影响。为此,我们提出LayerIF,一种数据驱动框架,利用影响函数以原则性且任务敏感的方式量化各层训练质量。通过分离每层梯度,计算验证损失对训练样本的敏感度,获得层级重要性估计。该方法能为同一模型生成任务特异的层重要性评分,揭示层在不同任务中的专业化特征。实验表明,基于影响函数的模型无关分配策略,在多个模型架构上均实现任务性能持续提升,有效应用于LoRA-MoE专家分配与层级稀疏性分布剪枝。

原文摘要 · Abstract (English)

Pretrained Large Language Models (LLMs) achieve strong performance across a wide range of tasks, yet exhibit substantial variability in the various layers' training quality with respect to specific downstream applications, limiting their downstream performance. It is therefore critical to estimate layer-wise training quality in a manner that accounts for both model architecture and training data. However, existing approaches predominantly rely on model-centric heuristics (such as spectral statistics, outlier detection, or uniform allocation) while overlooking the influence of data. To address these limitations, we propose LayerIF, a data-driven framework that leverages Influence Functions to quantify the training quality of individual layers in a principled and task-sensitive manner. By isolating each layer's gradients and measuring the sensitivity of the validation loss to training examples by computing layer-wise influences, we derive data-driven estimates of layer importance. Notably, our method produces task-specific layer importance estimates for the same LLM, revealing how layers specialize for different test-time evaluation tasks. We demonstrate the utility of our scores by leveraging them for two downstream applications: (a) expert allocation in LoRA-MoE architectures and (b) layer-wise sparsity distribution for LLM pruning. Experiments across multiple LLM architectures demonstrate that our model-agnostic, influence-guided allocation leads to consistent gains in task performance.

大模型影响函数层质量剪枝

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。