用冻结模型给对话添加说话人特征标签，无需微调即可高效实现。
Enhancing Dialogue Annotation with Speaker Characteristics Leveraging a Frozen LLM
- 用冻结的音频和语言模型结合，通过轻量连接器推断说话人特征。
- 在说话人画像任务上表现媲美专用模型，且保持快速与模块化。
- 可直接比较语音嵌入，实现8.8%等错误率，适合语音分析场景。
在对话转录流程中，大语言模型常用于后处理以提升语法、标点和可读性。本文探索一种互补的后处理步骤：通过添加说话人特征标签（如年龄、性别、情绪）来丰富转录内容，部分标签为全对话全局属性，部分为时变属性。方法结合冻结的音频基础模型（如Whisper或WavLM）与冻结的LLAMA语言模型，无需对任一模型进行任务特定微调。通过轻量高效的连接器融合音视频与语言表征，在说话人特征识别任务上取得具有竞争力的表现，同时保持系统模块化与处理速度。此外，我们证明冻结的LLAMA模型可直接比较x-vectors，在某些场景下达到8.8%的等错误率。
原文摘要 · Abstract (English)
In dialogue transcription pipelines, Large Language Models (LLMs) are frequently employed in post-processing to improve grammar, punctuation, and readability. We explore a complementary post-processing step: enriching transcribed dialogues by adding metadata tags for speaker characteristics such as age, gender, and emotion. Some of the tags are global to the entire dialogue, while some are time-variant. Our approach couples frozen audio foundation models, such as Whisper or WavLM, with a frozen LLAMA language model to infer these speaker attributes, without requiring task-specific fine-tuning of either model. Using lightweight, efficient connectors to bridge audio and language representations, we achieve competitive performance on speaker profiling tasks while preserving modularity and speed. Additionally, we demonstrate that a frozen LLAMA model can compare x-vectors directly, achieving an Equal Error Rate of 8.8% in some scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。