ReverBERT用状态空间模型实现高效文本驱动语音风格迁移。
ReverBERT: A State Space Model for Efficient Text-Driven Speech Style Transfer
- 基于状态空间模型与傅里叶变换，实现语音风格平滑调控。
- 推理速度远超基线，自然度与表现力均显著提升。
- 适合语音合成、智能语音助手等需要快速风格转换场景。
文本驱动语音风格迁移旨在根据文本描述调整语音的语调、语速和音色。现有方法多依赖大规模神经网络或预训练语言模型，计算开销大。本文提出ReverBERT，一种基于状态空间模型（SSM）的高效框架，受图像风格迁移方法启发，但专用于语音域。通过引入潜语音特征的离散傅里叶变换，实现连续平滑的风格调节。我们还设计了新型基于Transformer的SSM层，有效连接文本风格描述与声学特征，在大幅降低推理时间的同时保持高质量语音。在多个基准语音数据集上的实验表明，ReverBERT在自然度、表现力和计算效率上均显著优于基线方法。模型与代码已公开，以推动该领域研究。
原文摘要 · Abstract (English)
Text-driven speech style transfer aims to mold the intonation, pace, and timbre of a spoken utterance to match stylistic cues from text descriptions. While existing methods leverage large-scale neural architectures or pre-trained language models, the computational costs often remain high. In this paper, we present \emph{ReverBERT}, an efficient framework for text-driven speech style transfer that draws inspiration from a state space model (SSM) paradigm, loosely motivated by the image-based method of Wang and Liu~\cite{wang2024stylemamba}. Unlike image domain techniques, our method operates in the speech space and integrates a discrete Fourier transform of latent speech features to enable smooth and continuous style modulation. We also propose a novel \emph{Transformer-based SSM} layer for bridging textual style descriptors with acoustic attributes, dramatically reducing inference time while preserving high-quality speech characteristics. Extensive experiments on benchmark speech corpora demonstrate that \emph{ReverBERT} significantly outperforms baselines in terms of naturalness, expressiveness, and computational efficiency. We release our model and code publicly to foster further research in text-driven speech style transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。