发现语音模型中负责音素、性别、音高属性的神经元，可精准剪枝与编辑。
Property Neurons in Self-Supervised Speech Transformers
- 通过分析前馈层神经元，定位特定语音属性的专属神经元。
- 移除特定属性神经元后下游任务性能显著下降，证明其关键作用。
- 保护这些神经元剪枝比传统方法更高效，适合模型精简与编辑。
自监督语音Transformer的层间分析已有诸多研究，但尚缺乏能精确识别负责特定语音属性（如音素、性别、音高）的神经元子集的方法，以支持模型剪枝与编辑。本文在Transformer的前馈层中识别出一组属性神经元，用于研究语音属性的存储机制。实验表明，移除某类属性的神经元会导致相应下游任务性能显著下降，证明其重要性。将该方法应用于前馈层剪枝（占模型大部分参数），结果显示：剪枝时保护属性神经元比基于范数的剪枝方法显著更有效。相关代码已开源：https://github.com/nervjack2/PropertyNeurons。
原文摘要 · Abstract (English)
There have been many studies on analyzing self-supervised speech Transformers, in particular, with layer-wise analysis. It is, however, desirable to have an approach that can pinpoint exactly a subset of neurons that is responsible for a particular property of speech, being amenable to model pruning and model editing. In this work, we identify a set of property neurons in the feedforward layers of Transformers to study how speech-related properties, such as phones, gender, and pitch, are stored. When removing neurons of a particular property (a simple form of model editing), the respective downstream performance significantly degrades, showing the importance of the property neurons. We apply this approach to pruning the feedforward layers in Transformers, where most of the model parameters are. We show that protecting property neurons during pruning is significantly more effective than norm-based pruning. The code for identifying property neurons is available at https://github.com/nervjack2/PropertyNeurons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。