大模型首次实现多说话人场景下按指令精准转录语音
Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
- 用语音编码器提取说话人特征与语义信息，输入微调大模型
- 在多人对话场景中，按性别、顺序、语言等指令准确转录
- 适合需要灵活语音解析的智能助手与会议记录应用
大语言模型在语音任务中进展迅速，但多说话人场景下的应用仍待探索。本文提出MT-LLM系统，首次研究大模型在多说话人场景中根据多样化指令进行语音转录的能力，涵盖多说话人自动语音识别（ASR）、目标说话人识别及基于性别、出现顺序、语言、关键词等属性的语音识别。系统采用WavLM和Whisper编码器提取多维度语音表征，结合LoRA微调的大模型实现语音理解与转录。实验表明，在鸡尾酒会场景中表现优异，验证了大模型在复杂环境中依据用户指令处理语音任务的潜力。代码、模型与示例已开源。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities. Despite progress in speech-related tasks, LLMs have not been sufficiently explored in multi-talker scenarios. In this work, we present a pioneering effort to investigate the capability of LLMs in transcribing speech in multi-talker environments, following versatile instructions related to multi-talker automatic speech recognition (ASR), target talker ASR, and ASR based on specific talker attributes such as sex, occurrence order, language, and keyword spoken. Our approach utilizes WavLM and Whisper encoder to extract multi-faceted speech representations that are sensitive to speaker characteristics and semantic context. These representations are then fed into an LLM fine-tuned using LoRA, enabling the capabilities for speech comprehension and transcription. Comprehensive experiments reveal the promising performance of our proposed system, MT-LLM, in cocktail party scenarios, highlighting the potential of LLM to handle speech-related tasks based on user instructions in such complex settings. The code, model, and samples are available at https://github.com/cuhealthybrains/MT-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。