用稀疏自编码器解析语音模型内部表示，发现能有效分离笑声、耳语等声音特征。
AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders
- 在Whisper和HuBERT各层训练稀疏自编码器，提取可解释的音频特征。
- 50%以上特征跨随机种子稳定，仅移除19%-27%即可消除特定概念。
- 可减少语音识别误检70%，且与人类脑电活动高度相关，适合研究听觉认知。
稀疏自编码器（SAEs）是解析神经表征的强大工具，但在音频领域应用仍不充分。本文在Whisper和HuBERT所有编码器层上训练SAEs，全面评估其稳定性、可解释性并验证实际效用。超过50%的特征在不同随机种子下保持一致，重建质量不受影响。SAE特征捕捉通用声学与语义信息，以及具体事件，如环境噪声、副语言声音（如笑声、耳语），并能有效解耦，仅需移除19%-27%特征即可消除某一概念。特征操控使Whisper的误检率降低70%，且单词错误率增加极小，体现实际应用价值。此外，SAE特征与人类语音感知时的脑电活动存在相关性，表明其与人脑处理机制对齐。代码与检查点已开源。
原文摘要 · Abstract (English)
Sparse Autoencoders (SAEs) are powerful tools for interpreting neural representations, yet their use in audio remains underexplored. We train SAEs across all encoder layers of Whisper and HuBERT, provide an extensive evaluation of their stability, interpretability, and show their practical utility. Over 50% of the features remain consistent across random seeds, and reconstruction quality is preserved. SAE features capture general acoustic and semantic information as well as specific events, including environmental noises and paralinguistic sounds (e.g. laughter, whispering) and disentangle them effectively, requiring removal of only 19-27% of features to erase a concept. Feature steering reduces Whisper's false speech detections by 70% with negligible WER increase, demonstrating real-world applicability. Finally, we find SAE features correlated with human EEG activity during speech perception, indicating alignment with human neural processing. The code and checkpoints are available at https://github.com/audiosae/audiosae_demo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。