arXiv:2505.13085eess.AScs.LG2025-05中稿 · Interspeech 2025被引 4

用通用语音编码器分离语义与说话人特征,保护隐私同时保留语音内容。

Universal Semantic Disentangled Privacy-preserving Speech Representation Learning

  • 通过通用语音编码器将语音拆分为语义信息和说话人特征两部分
  • 语义表示保留内容、语调和情感,且不泄露说话人身份
  • 适合需要语音隐私保护的大型模型训练场景

使用人类语音录音训练大语言模型存在隐私风险,因模型可能生成与训练数据高度相似的输出。本文提出一种基于通用语音编码器(USC)的说话人隐私保护表征学习方法,该模型计算高效,能将语音解耦为:(i) 隐私保护的语义丰富表示,包含内容与副语言信息;(ii) 剩余声学与说话人表示,用于高保真重建。大量评估表明,USC的语义表示保留了内容、语调和情感,同时去除可识别的说话人属性。结合两类表示,USC实现了当前最优的语音重建性能。我们还引入了一种符合感知测试的隐私保护评估方法,对比了文献中其他编码器,验证了USC在说话人匿名化、副语言信息保留与内容完整性之间的有效权衡。音频样本见 https://www.amazon.science/usc-samples。

原文摘要 · Abstract (English)

The use of audio recordings of human speech to train LLMs poses privacy concerns due to these models' potential to generate outputs that closely resemble artifacts in the training data. In this study, we propose a speaker privacy-preserving representation learning method through the Universal Speech Codec (USC), a computationally efficient encoder-decoder model that disentangles speech into: (i) privacy-preserving semantically rich representations, capturing content and speech paralinguistics, and (ii) residual acoustic and speaker representations that enables high-fidelity reconstruction. Extensive evaluations presented show that USC's semantic representation preserves content, prosody, and sentiment, while removing potentially identifiable speaker attributes. Combining both representations, USC achieves state-of-the-art speech reconstruction. Additionally, we introduce an evaluation methodology for measuring privacy-preserving properties, aligning with perceptual tests. We compare USC against other codecs in the literature and demonstrate its effectiveness on privacy-preserving representation learning, illustrating the trade-offs of speaker anonymization, paralinguistics retention and content preservation in the learned semantic representations. Audio samples are shared in https://www.amazon.science/usc-samples.

语音隐私语义解耦编码器大模型训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。