arXiv:2509.19592eess.AScs.AI2025-09中稿 · ICASSP 2026

用局部变压器提升语音生成速度,不损失音质。

Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generation

  • 设计两种局部变换器,分步或掩码预测多代码本
  • 帧堆叠技术实现多帧联合预测,提速显著
  • 提供不同场景下策略选择指南,实用性强

基于大语言模型的语音生成通常在离散声学代码上运行,其多代码本结构与文本标记有本质差异。每个时间步需联合预测N个代码本条目,引入依赖关系,挑战简单并行预测方法。并行预测虽高效但常牺牲保真度。为此,层次化策略采用局部变换器(LT)捕捉时间步内依赖关系。本文系统研究两种LT架构:自回归变换器逐个生成代码本,以及基于MaskGIT的迭代掩码预测变换器。两者均支持帧堆叠,主变换器联合预测多帧,LT解码代码本,显著提升速度且不降低感知质量。通过大量分析,量化了并行与迭代采样在不同吞吐量和质量区间下的权衡。最终提出根据部署需求(如计算效率与合成保真度)选择解码策略的实用指南。

原文摘要 · Abstract (English)

Speech generation models based on large language models (LLMs) typically operate on discrete acoustic codes, which differ fundamentally from text tokens due to their multicodebook structure. At each timestep, models must predict N codebook entries jointly, introducing dependencies that challenge simple parallel prediction approaches. Parallel prediction assumes independence among codebooks, yielding efficient decoding but often at the cost of reduced fidelity. To address this, hierarchical strategies employ a local transformer (LT) to refine predictions and capture intra-timestep dependencies. In this work, we systematically investigate two LT architectures: an autoregressive transformer that generates codebooks sequentially, and a MaskGIT-based transformer that performs iterative masked prediction. Both designs further enable frame stacking, where the primary transformer predicts multiple frames jointly, and the LT decodes their codebooks, offering improvements in speed without compromising perceptual quality. Through extensive analysis, we characterize the tradeoffs between parallel and iterative sampling strategies across different throughput and quality regimes. Finally, we propose practical guidelines for selecting decoding strategies based on deployment priorities such as computational efficiency and synthesis fidelity.

语音生成局部变换器代码本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。