发现预训练模型中存在通用最优子网，可高效微调。
SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks
- 从谱平衡与任务能量角度证明子网有效性
- 仅更新部分权重即达主流微调效果
- 适合追求高效、轻量微调的工程应用
本文提出理论框架，解释为何在预训练模型中随机选取的小规模子网络（切片）进行微调即可满足下游任务需求。我们证明预训练模型具有普遍最优切片特性，源于两个现象：(1) 谱平衡——不同权重矩阵切片的特征谱分布高度相似；(2) 高任务能量——主干特征保留丰富任务相关信息。由此提出通用最优切片假说，为大规模模型参数高效微调（PEFT）提供理论基础。受此启发，我们提出SliceFine方法，通过仅更新原始权重的选定切片实现微调，不引入额外参数，区别于适配器类方法。实证表明，SliceFine在语言与视觉任务上性能媲美当前最佳PEFT方法，同时显著提升训练速度、内存效率与模型紧凑性。本工作连接理论与实践，为现有PEFT技术提供了理论支持的新路径。
原文摘要 · Abstract (English)
This paper presents a theoretical framework explaining why fine tuning small, randomly selected subnetworks (slices) within pre trained models can be sufficient for downstream adaptation. We prove that pretrained networks exhibit a universal winning slice property arising from two phenomena: (1) spectral balance the eigenspectra of different weight matrix slices are remarkably similar; and (2) high task energy their backbone representations retain rich, task relevant features. This leads to the Universal Winning Slice Hypothesis, which provides a theoretical foundation for parameter efficient fine tuning (PEFT) in large scale models. Inspired by this, we propose SliceFine, a PEFT method that exploits this inherent redundancy by updating only selected slices of the original weights introducing zero new parameters, unlike adapter-based approaches. Empirically, SliceFine matches the performance of state of the art PEFT methods across language and vision tasks, while significantly improving training speed, memory efficiency, and model compactness. Our work bridges theory and practice, offering a theoretically grounded alternative to existing PEFT techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。