评测大模型在多人对话中识别说话人能力的基准
M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models
- 基于四个公开数据集构建1.2万+对话实例的多模态评测集
- 模型能理解内容却常错判说话人，暴露对话感知短板
- 适合研究多说话人语音理解、对话系统与大模型评估者
我们提出M3-SLU，一个用于评估多说话人、多轮口语理解的新型多模态大模型基准。尽管当前模型在语音和文本理解方面表现优异，但在说话人归因推理（即理解谁在何时说了什么）方面仍存在不足。M3-SLU整合了四个公开语料库（CHiME-6、MELD、MultiDialog 和 AMI），包含超过12,000个经验证的实例，配套音频、转录文本与元数据。该基准包含两项任务：(1) 说话人归因问答，(2) 通过话语匹配进行说话人归因。我们提供了级联流水线与端到端多模态大模型的基线结果，采用大语言模型作为裁判与准确率指标进行评估。结果显示，模型虽能正确捕捉所说内容，但常无法准确识别说话人，揭示了对话理解中说话人感知的关键差距。M3-SLU为推进说话人感知的多模态理解研究提供了一个具有挑战性的基准。
原文摘要 · Abstract (English)
We present M3-SLU, a new multimodal large language model (MLLM) benchmark for evaluating multi-speaker, multi-turn spoken language understanding. While recent models show strong performance in speech and text comprehension, they still struggle with speaker-attributed reasoning, the ability to understand who said what and when in natural conversations. M3-SLU is built from four open corpora (CHiME-6, MELD, MultiDialog, and AMI) and comprises over 12,000 validated instances with paired audio, transcripts, and metadata. It includes two tasks: (1) Speaker-Attributed Question Answering and (2) Speaker Attribution via Utterance Matching. We provide baseline results for both cascaded pipelines and end-to-end MLLMs, evaluated using an LLM-as-Judge and accuracy metrics. Results show that while models can capture what was said, they often fail to identify who said it, revealing a key gap in speaker-aware dialogue understanding. M3-SLU offers as a challenging benchmark to advance research in speaker-aware multimodal understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。