通过缓存Transformer层输出,加速扩散TTS模型推理,无需重训练。
Accelerating Diffusion Transformer-Based Text-to-Speech with Transformer Layer Caching
- 引入SmoothCache,只缓存自注意力与前馈层输出,减少去噪重复计算。
- 高阶去噪步骤缓存可提速30%以上,且音质无损;低阶缓存则影响质量。
- 统一缓存策略适配多层结构,适合部署在资源受限场景的TTS系统。
本文提出一种加速基于扩散Transformer(DiT)的文本转语音(TTS)模型推理的方法,通过选择性缓存机制实现。具体地,将SmoothCache集成至F5-TTS架构中,聚焦于缓存自注意力与前馈网络层的输出,以减少去噪过程中的冗余计算。引入校准阶段,分析不同时步间的L1相对误差,指导缓存策略选择,以最小化质量下降。为解决层间依赖问题,采用统一缓存方案,将自注意力层推导出的缓存模式应用于两类层。在LibriSpeech-PC和Seed-TTS数据集上评估不同缓存阈值与去噪步数配置。结果表明,高阶去噪步骤缓存可显著降低推理时间,且不影响输出质量;而低阶缓存会像减少总去噪步数一样损害合成质量。客观与主观评价证实SmoothCache在保持性能的同时提升计算效率。缓存推理与少步推理的对比进一步凸显选择性缓存的优势,尤其在高步数配置下。本工作证明,变压器层缓存是优化扩散变压器型TTS模型的实用方案,无需架构修改或重新训练。示例合成结果见:https://siratish.github.io/F5-TTS_SmoothCache/
原文摘要 · Abstract (English)
This paper presents a method to accelerate the inference process of diffusion transformer (DiT)-based text-to-speech (TTS) models by applying a selective caching mechanism to transformer layers. Specifically, I integrate SmoothCache into the F5-TTS architecture, focusing on caching outputs of self-attention and feed-forward network layers to reduce redundant computations during the denoising process. A calibration phase is introduced to analyze L1 relative errors between timesteps, guiding the selection of cache schedules that minimize quality degradation. To address the problem of inter-layer dependency, a unified caching schedule is adopted, applying the cache pattern derived from self-attention layers to both layer types. Experiments on LibriSpeech-PC and Seed-TTS datasets evaluate various cache thresholds and denoising step configurations. Results show that caching at higher denoising steps reduces inference time without compromising output quality, whereas caching at lower steps can negatively impact synthesis quality similarly to reducing the total number of denoising steps. Objective and subjective metrics confirm the effectiveness of SmoothCache in maintaining performance while improving computational efficiency. Comparisons between cached inference and reduced-step inference further highlight the benefits of selective caching, especially under high-step configurations. This work demonstrates that transformer layer caching is a practical solution for optimizing diffusion transformer-based TTS models without requiring architectural changes or retraining. Example inference results can be heard at https://siratish.github.io/F5-TTS_SmoothCache/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。