一个模型搞定语音识别、合成与转换，还能在低延迟设备上运行。
Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers
- 用统一的自回归框架处理语音任务，共享离散音频令牌空间。
- 0.3亿参数轻量版可部署在边缘设备，性能媲美专用模型。
- 支持指令驱动任务切换，无需修改结构，适合实际应用。
传统语音系统分别使用独立模型完成文本转语音（TTS）、自动语音识别（ASR）和语音转换（VC），导致流程割裂，难以扩展、效率低下且跨任务泛化能力差。本文提出通用音频基础模型GPA，将多个核心语音任务集成于单一大语言模型架构中。GPA基于共享的离散音频令牌空间，支持指令驱动的任务诱导，使单个自回归模型无需结构修改即可灵活执行TTS、ASR与VC。该统一设计结合了全自回归离散语音令牌建模、跨语音域联合多任务训练，以及高并发高吞吐的可扩展推理管道。所提出的模型家族支持多尺度部署，包含一个专为边缘和资源受限环境优化的0.3亿参数轻量版本。这些设计表明，统一的自回归架构可在多种语音任务上实现竞争性性能，同时具备低延迟、实用部署的可行性。
原文摘要 · Abstract (English)
Traditional speech systems typically rely on separate, task-specific models for text-to-speech (TTS), automatic speech recognition (ASR), and voice conversion (VC), resulting in fragmented pipelines that limit scalability, efficiency, and cross-task generalization. In this paper, we present General-Purpose Audio (GPA), a unified audio foundation model that integrates multiple core speech tasks within a single large language model (LLM) architecture. GPA operates on a shared discrete audio token space and supports instruction-driven task induction, enabling a single autoregressive model to flexibly perform TTS, ASR, and VC without architectural modifications. This unified design combines a fully autoregressive formulation over discrete speech tokens, joint multi-task training across speech domains, and a scalable inference pipeline that achieves high concurrency and throughput. The resulting model family supports efficient multi-scale deployment, including a lightweight 0.3B-parameter variant optimized for edge and resource-constrained environments. Together, these design choices demonstrate that a unified autoregressive architecture can achieve competitive performance across diverse speech tasks while remaining viable for low-latency, practical deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。