测试语音模型能否通过重音差异理解语义，发现现有模型表现不佳。
StressTest: Can YOUR Speech LM Handle the Stress?
- 构建重音差异基准 StressTest，评估模型对语音语义的理解能力。
- 在真实录音上，新模型 StresSLM 显著优于现有语音语言模型。
- 提出数据生成方法，创建包含 17,000 条重音变化样本的 Stress-17k 数据集。
句法重音指在口语中对某些词语施加重音以突出或对比某种意义，常用于表达未明说的意图。近年来，语音感知语言模型（SLMs）实现了直接音频处理，使模型能够利用语音的完整信息完成语音推理任务，如口语问答。然而，尽管重音在塑造语义和意图中起关键作用，却在 SLM 的评估与开发中被严重忽视。为此，本文提出 StressTest 基准，用于评估模型基于重音模式区分语义的能力。我们评估了主流 SLMs，发现它们虽整体能力强，但在重音相关任务上表现不佳。因此，我们设计了一种新颖的数据生成流程，构建了 Stress-17k，一个模拟重音变化导致语义改变的训练数据集。结果表明，经微调后的 StresSLM 模型在真实录音上具有良好泛化能力，并显著优于现有 SLMs 在重音推理与检测任务上的表现。相关模型、代码、数据与样例可访问：pages.cs.huji.ac.il/adiyoss-lab/stresstest。
原文摘要 · Abstract (English)
Sentence stress refers to emphasis on words within a spoken utterance to highlight or contrast an idea. It is often used to imply an underlying intention not explicitly stated. Recent speech-aware language models (SLMs) have enabled direct audio processing, allowing models to access the full richness of speech to perform audio reasoning tasks such as spoken question answering. Despite the crucial role of sentence stress in shaping meaning and intent, it remains largely overlooked in evaluation and development of SLMs. We address this gap by introducing StressTest, a benchmark designed to evaluate models' ability to distinguish between meanings of speech based on the stress pattern. We evaluate leading SLMs, and find that despite their overall capabilities, they perform poorly on such tasks. Hence, we propose a novel data generation pipeline, and create Stress-17k, a training set that simulates change of meaning implied by stress variation. Results suggest, that our finetuned model, StresSLM, generalizes well to real recordings and notably outperforms existing SLMs on sentence stress reasoning and detection. Models, code, data, samples - pages.cs.huji.ac.il/adiyoss-lab/stresstest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。