2.3B参数的LLM语音识别框架,兼顾效率与抗噪能力。
NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR

- 分模块设计编码器与LLM功能边界,优化多阶段训练流程
- 仅用2.3亿参数达到顶尖性能,内测中实体识别显著领先
- 支持百万级关键词定制,毫秒级检索,适合实时部署
将大语言模型(LLM)融入自动语音识别(ASR)已成为近年主流范式。尽管现有基于LLM的ASR模型在公开基准上表现优异,但其训练仍以数据驱动为主,未能充分解决实际部署中的关键挑战——资源受限场景下的可扩展性不足,以及声学困难条件下的幻觉问题。为此,我们提出NIM4-ASR,一个面向生产环境的高效、鲁棒的基于LLM的ASR框架。基于编码器与LLM的功能角色划分,我们重构了多阶段训练范式,使各模块与其能力边界对齐:重新设计预训练架构与目标,缓解模态差距并提升参数效率;引入迭代异步监督微调(SFT)阶段,保持声学保真度并控制表示漂移;设计专用于ASR的强化学习阶段,进一步提升识别质量与鲁棒性。此外,集成一系列生产优化:噪声与静音下稳健表现、实时流式推理、通过检索增强生成(RAG)实现热词定制。实验表明,NIM4-ASR在多个公开基准上以仅2.3亿参数达到业界顶尖水平,且在内部基准上显著优于更大规模模型,尤其在实体密集的真实场景中表现突出。该框架还支持通过RAG实现百万级热词定制,检索延迟低于1毫秒,可高效适应新实体与个性化需求。
原文摘要 · Abstract (English)
Integrating large language models (LLMs) into automatic speech recognition (ASR) has become a mainstream paradigm in recent years. Although existing LLM-based ASR models demonstrate impressive performance on public benchmarks, their training remains predominantly data-driven, leaving key practical challenges insufficiently addressed -- particularly limited downward scalability in resource-constrained deployments and hallucinations under acoustically challenging conditions. To address these issues, we present NIM4-ASR, a production-oriented LLM-based ASR framework optimized for both efficiency and robustness. Grounded in a principled delineation of functional roles between the encoder and the LLM, we redesign the multi-stage training paradigm to align each module with its intended capability boundary. Specifically, we reformulate the pre-training architecture and objective to mitigate the modality gap and improve parameter efficiency; introduce an iterative asynchronous SFT stage to preserve acoustic fidelity and constrain representation drift; and design an ASR-specialized reinforcement learning stage to further enhance recognition quality and robustness. We additionally incorporate a suite of production-oriented optimizations, including robustness under noisy and silent conditions, real-time streaming inference, and hotword customization via retrieval-augmented generation (RAG). Experiments show that NIM4-ASR achieves state-of-the-art performance on multiple public benchmarks with merely 2.3B parameters, while substantially outperforming larger-scale competitors on internal benchmarks -- particularly in entity-intensive real-world scenarios. NIM4-ASR further supports million-scale hotword customization via RAG with sub-millisecond retrieval latency, enabling efficient adaptation to emerging entities and personalized user requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。