用语音感知大模型实现更准的说话人归属语音识别
Speaker Attributed Automatic Speech Recognition Using Speech Aware LLMS
- 在大模型中引入说话人聚类标签，联合训练提升识别精度
- 通过人工拼接多说话人对话增强数据，缓解训练数据不足
- 直接端到端处理，比先分说话人再识别更高效准确
说话人归属自动语音识别（SAA）通过在转录文本中加入说话人身份标签（如[Speaker 1]:）增强传统ASR系统。本文扩展了最先进的语音感知大语言模型Granite-speech的能力，该模型原用于转录与翻译。仅需少量架构修改，即可有效适配SAA任务。核心贡献是引入说话人聚类识别标签（如[Speaker 1 cluster 42]:），并与SAA联合训练，显著提升准确率。针对训练数据有限的问题，提出一种数据增强方法：人工拼接多说话人对话。所提方法在多个基准上评估，性能优于先进行说话人分割再识别的传统流水线。
原文摘要 · Abstract (English)
Speaker-Attributed Automatic Speech Recognition (SAA) enhances traditional ASR systems by incorporating relative speaker identity tags directly into the transcript (e.g., [Speaker 1]:, [Speaker 2]:). In this work, we extend the capabilities of Granite-speech, a state-of-the-art speech-aware Large Language Model (LLM) originally trained for transcription and translation. We demonstrate that it can be effectively adapted for SAA with only minimal architectural changes. Our core contribution is the introduction of speaker cluster identification tags (e.g., [Speaker 1 cluster 42]:) which are jointly trained with SAA to significantly improve accuracy. To address limitations in training data, we propose a data augmentation method that uses artificially concatenated multi-speaker conversations. Our approach is evaluated across multiple benchmarks and shows superior performance compared to conventional pipelines that sequentially perform speaker diarization followed by ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。