轻量高效多语言语音模型，中文英文识别快10倍且准确率高。
Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST
- 用FastConformer+Transformer结构，1.7万小时数据训练，抗干扰强。
- 英语识别比Whisper-large快10倍，多语言性能媲美大模型。
- 适合低资源部署，也适用于实时语音转写与翻译场景。
本文介绍Canary-1B-v2，一种快速、鲁棒的多语言自动语音识别（ASR）与语音转文字翻译（AST）模型。采用FastConformer编码器和Transformer解码器，支持25种主要为欧洲语言。模型在总计170万小时的数据样本上训练，包括Granary和NeMo ASR Set 3.0，并加入非语音音频以减少幻觉。描述了两阶段预训练与微调流程，结合动态数据平衡，并实验了nGPT编码器。结果表明，nGPT在大规模数据下表现良好，而FastConformer在微调后更优。时间戳使用NeMo强制对齐器（NFA）与辅助CTC模型，提供可靠的段级时间标记。评估显示，Canary-1B-v2在英语ASR上优于Whisper-large-v3，同时速度快10倍；多语言ASR与AST性能可媲美Seamless-M4T-v2-large和基于LLM的系统。此外，发布其升级版Parakeet-TDT-0.6B-v3，仅6亿参数，同样覆盖25种语言，实现轻量化多语言ASR。
原文摘要 · Abstract (English)
This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encoder and Transformer decoder, it supports 25 languages primarily European. The model was trained on 1.7M hours of total data samples, including Granary and NeMo ASR Set 3.0, with non-speech audio added to reduce hallucinations for ASR and AST. We describe its two-stage pre-training and fine-tuning process with dynamic data balancing, as well as experiments with an nGPT encoder. Results show nGPT scales well with massive data, while FastConformer excels after fine-tuning. For timestamps, Canary-1B-v2 uses the NeMo Forced Aligner (NFA) with an auxiliary CTC model, providing reliable segment-level timestamps for ASR and AST. Evaluations show Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10x faster, and delivers competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large and LLM-based systems. We also release Parakeet-TDT-0.6B-v3, a successor to v2, offering multilingual ASR across the same 25 languages with just 600M parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。