用新注意力机制和初始状态调优，实现多样本语音克隆与风格控制。
Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesis
- 用门控线性注意力替代自注意力，提升推理速度。
- 支持任意长度和数量的语音样本输入，实现多样语音风格克隆。
- 适合需要精细控制语调、情感的语音合成应用。
基于Transformer的神经编解码语言模型已革新文本到语音合成，通过前缀续写任务实现语音克隆。然而其有限的上下文长度限制了对说话人语调和风格的覆盖范围与多样性。同时，从短前缀中适配语调、口音或情感仍具挑战。此外，自注意力的二次复杂度也制约了推理吞吐量。本文提出Lina-Speech，采用门控线性注意力（GLA）取代标准自注意力，作为核心架构，在保持顶尖性能的同时显著提升推理效率。结合循环结构的状态特性，引入初始状态调优（IST）策略，实现对任意数量和长度语音样本的多样本条件建模，提供全面高效的语音克隆及跨领域语音风格与情感适应方案。实验验证其对语调、情感等细粒度特征的有效控制能力。代码、模型权重及演示均已开源：https://github.com/theodorblackbird/lina-speech
原文摘要 · Abstract (English)
Neural codec language models, built on transformer architecture, have revolutionized text-to-speech (TTS) synthesis, excelling in voice cloning by treating it as a prefix continuation task. However, their limited context length hinders their effectiveness to short speech samples. As a result, the voice cloning ability is restricted to a limited coverage and diversity of the speaker's prosody and style. Besides, adapting prosody, accent, or appropriate emotion from a short prefix remains a challenging task. Finally, the quadratic complexity of self-attention limits inference throughput. In this work, we introduce Lina-Speech, a TTS model with Gated Linear Attention (GLA) to replace standard self-attention as a principled backbone, improving inference throughput while matching state-of-the-art performance. Leveraging the stateful property of recurrent architecture, we introduce an Initial-State Tuning (IST) strategy that unlocks the possibility of multiple speech sample conditioning of arbitrary numbers and lengths and provides a comprehensive and efficient strategy for voice cloning and out-of-domain speaking style and emotion adaptation. We demonstrate the effectiveness of this approach for controlling fine-grained characteristics such as prosody and emotion. Code, checkpoints, and demo are freely available: https://github.com/theodorblackbird/lina-speech
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。