arXiv:2412.15726cs.CLeess.AS2024-12被引 19

用句子级数据生成长音频训练集,提升低资源语言语音转文字效果

Fine-tuning Whisper on Low-Resource Languages for Real-World Applications

  • 将易获取的句子级数据转化为长文本语料,支持模型处理长音频
  • 在瑞士德语上达到新SOTA,BLEU分数显著高于未微调Whisper
  • 方法可复用于其他低资源语言,仅需句子级数据即可实现高质量转写

本文提出一种针对低资源语言的Whisper模型微调新方法,以瑞士德语为例,通过将易获取的句子级数据转换为长文本语料,克服了非句子级数据难获取且受版权限制的问题。该方法在不依赖长音频数据的前提下,保持了模型对长音频的处理与分割能力。实验表明,该方法显著提升了多个真实应用场景下的性能,使瑞士德语语音转文字(STT)模型达到新SOTA。相比未微调的Whisper及此前最优模型,新模型在BLEU指标上表现更优。研究还证明该方法适用于其他低资源语言,附带代码和文档,支持仅用句子级数据构建高质量、具备长音频转写能力的微调Whisper模型。

原文摘要 · Abstract (English)

This paper presents a new approach to fine-tuning OpenAI's Whisper model for low-resource languages by introducing a novel data generation method that converts sentence-level data into a long-form corpus, using Swiss German as a case study. Non-sentence-level data, which could improve the performance of long-form audio, is difficult to obtain and often restricted by copyright laws. Our method bridges this gap by transforming more accessible sentence-level data into a format that preserves the model's ability to handle long-form audio and perform segmentation without requiring non-sentence-level data. Our data generation process improves performance in several real-world applications and leads to the development of a new state-of-the-art speech-to-text (STT) model for Swiss German. We compare our model with a non-fine-tuned Whisper and our previous state-of-the-art Swiss German STT models, where our new model achieves higher BLEU scores. Our results also indicate that the proposed method is adaptable to other low-resource languages, supported by written guidance and code that allows the creation of fine-tuned Whisper models, which keep segmentation capabilities and allow the transcription of longer audio files using only sentence-level data with high quality.

语音识别低资源语言Whisper微调数据增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。