让自回归语音合成模型实现恒定计算量，大幅降低内存占用。
WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
- 用滑动窗口注意力替代全序列自注意，分块处理生成语音。
- 实验显示最高可节省66.2%的缓存内存，每步延迟几乎不变。
- 适合需要低延迟、高效率语音合成的应用场景。
近期的解码器仅自回归语音合成（AR-TTS）模型能生成高质量语音，但其内存与计算开销随序列长度呈平方增长，源于全自注意力机制。本文提出WAND框架，将注意力分为两部分：对条件标记使用持久全局注意力，对生成标记采用局部滑动窗口注意力。为稳定微调，采用渐进收紧注意力窗口的课程学习策略。进一步通过从全注意力教师模型中进行知识蒸馏，以极高的数据效率恢复高质量语音合成效果。在三个现代AR-TTS模型上评估，WAND在保持原始音质的同时，实现最高达66.2%的键值缓存内存减少，且每步延迟基本恒定。
原文摘要 · Abstract (English)
Recent decoder-only autoregressive text-to-speech (AR-TTS) models produce high-fidelity speech, but their memory and compute costs scale quadratically with sequence length due to full self-attention. In this paper, we propose WAND, Windowed Attention and Knowledge Distillation, a framework that adapts pretrained AR-TTS models to operate with constant computational and memory complexity. WAND separates the attention mechanism into two: persistent global attention over conditioning tokens and local sliding-window attention over generated tokens. To stabilize fine-tuning, we employ a curriculum learning strategy that progressively tightens the attention window. We further utilize knowledge distillation from a full-attention teacher to recover high-fidelity synthesis quality with high data efficiency. Evaluated on three modern AR-TTS models, WAND preserves the original quality while achieving up to 66.2% KV cache memory reduction and length-invariant, near-constant per-step latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。