通过噪声增强提升连续自回归模型的生成质量，避免误差累积。
Continuous Autoregressive Models with Noise Augmentation Avoid Error Accumulation
- 训练时注入随机噪声，让模型对推理误差更鲁棒。
- 在长序列生成中保持音频质量，显著优于传统自回归与非自回归方法。
- 适合实时交互式生成场景，如音乐创作与语音合成。
自回归模型通常用于离散标记序列，但近期研究表明以自回归方式生成连续嵌入也是可行的。然而，此类连续自回归模型（CAM）在长序列生成中常因推理过程中的误差累积导致质量下降。本文提出一种新方法：在训练时向输入嵌入注入随机噪声，使模型对不同水平的误差具有鲁棒性；进一步在推理阶段引入低层噪声，有效减少误差累积。在音乐音频生成任务上的实验表明，CAM 显著优于现有自回归与非自回归方法，在长序列下仍能保持高质量音频输出。该工作为纯自回归生成连续嵌入提供了新路径，拓展了实时与交互式生成应用的可能性。
原文摘要 · Abstract (English)
Autoregressive models are typically applied to sequences of discrete tokens, but recent research indicates that generating sequences of continuous embeddings in an autoregressive manner is also feasible. However, such Continuous Autoregressive Models (CAMs) can suffer from a decline in generation quality over extended sequences due to error accumulation during inference. We introduce a novel method to address this issue by injecting random noise into the input embeddings during training. This procedure makes the model robust against varying error levels at inference. We further reduce error accumulation through an inference procedure that introduces low-level noise. Experiments on musical audio generation show that CAM substantially outperforms existing autoregressive and non-autoregressive approaches while preserving audio quality over extended sequences. This work paves the way for generating continuous embeddings in a purely autoregressive setting, opening new possibilities for real-time and interactive generative applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。