针对孟加拉语长音频语音识别与说话人分离,构建了端到端的高效解决方案。
Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization

- 微调Whisper和PyAnnote模型,结合数据增强与完整权重训练
- 语音识别WER达0.2441,说话人分离DER为0.2392,显著优于基线
- 适用于孟加拉语语音处理研究者及多说话人场景应用开发
由于长时录音、多样声学环境和显著说话人差异,孟加拉语的自动语音识别(ASR)与说话人分离仍具挑战。本文针对这两个核心任务,构建了鲁棒的长时序语音识别与说话人分离系统。对于语音识别,基于约15,000段分块对齐的孟加拉语音频,使用full weight训练方式微调tugstugi bengaliai regional asr whisper medium模型,并引入噪声注入、混响模拟、回声、裁剪失真及音高/时间扰动等数据增强策略。对于说话人分离,基于竞赛标注数据集,使用PyTorch Lightning微调pyannote/segmentation-3.0模型,并将优化后的分割主干替换至pyannote/speaker-diarization-community-1流程中,保留预训练说话人嵌入与聚类组件。最终系统在测试集上分别取得0.2441的词错误率(WER)与0.2392的分离错误率(DER),显著优于各自预训练基线。本文详细描述了从数据预处理、文本归一化、音频增强、训练策略、推理优化到后处理的完整流程。
原文摘要 · Abstract (English)
Automatic Speech Recognition (ASR) and speaker diarization in Bangla remain challenging due to long form recordings, diverse acoustic conditions, and significant speaker variability. This work addresses these two core tasks in Bangla spoken language understanding by developing robust systems for long form ASR and speaker diarization. For ASR (Problem 1), we fine tune the tugstugi bengaliai regional asr whisper medium model on a custom-curated dataset of approximately 15,000 chunked and aligned Bangla audio segments, employing full weight training with extensive data augmentation including noise injection, reverb simulation, echo, clipping distortion, and pitch/time perturbation. For speaker diarization (Problem 2), we fine-tune the pyannote/segmentation-3.0 model using PyTorch Lightning on the competition annotated diarization dataset, swapping the fine-tuned segmentation backbone into the pyannote/speaker-diarization-community-1 pipeline while retaining the pretrained speaker embedding and clustering components. Our ASR system achieves a Word Error Rate (WER) of 0.2441, while our diarization system achieves a Diarization Error Rate (DER) of 0.2392, both evaluated on the test set, demonstrating notable improvements over the respective pretrained baselines. We describe our complete pipeline, including data preprocessing, text normalization, audio augmentation, training strategies, inference optimization, and post-processing for both tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。