ESPnet3提升语音研究可扩展性，支持大规模高效训练。
ESPnet3: Infrastructure for Scalable Speech and Audio Research in the Foundation Model Era
- 模块化架构+配置驱动，灵活整合多源数据集
- 比ESPnet2每轮训练快21.1分钟，多节点训练GPU利用率超80%
- 新增模型或数据仅需约46行代码，适合大团队协作
近期语音研究面临日益庞大的数据集、复杂的模型和多样化的实验流程，但现有框架需大量工程投入。我们提出ESPnet3，一个基于模块化系统架构的语音与音频研究框架，采用配置驱动的数据集组合和统一的Python工作流。ESPnet3引入DataOrganizer抽象实现灵活的数据集集成，并通过数据分片实现内存高效的规模化训练，同时支持通过轻量级阶段覆盖实现配方特异性逻辑。在OWSM预训练实验中，相比ESPnet2，ESPnet3将每轮训练时间减少21.1分钟，多节点训练中实现超过80%的GPU利用率。微调实验表明，新模型与数据集的集成仅需约46行额外代码。ESPnet3将公开发布，包含模型检查点与训练日志。
原文摘要 · Abstract (English)
Recent speech research involves increasingly large datasets, complex models, and diverse experimental workflows. However, existing frameworks require substantial engineering effort to support such experiments. We present ESPnet3, a speech and audio research framework built on a modular system architecture with configuration-driven dataset composition and unified Python-based workflows. ESPnet3 introduces a DataOrganizer abstraction for flexible dataset integration and dataset sharding for memory-efficient large-scale training, while allowing recipe-specific logic through lightweight stage overrides. In OWSM pre-training experiments, ESPnet3 reduces per-epoch training time by \emph{21.1 minutes} compared to ESPnet2 and achieves \emph{>80\% GPU utilization} in multi-node training. Fine-tuning experiments show that new models and datasets can be integrated with around \emph{46 lines of additional code}. ESPnet3 will be publicly released with model checkpoints and training logs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。