用检索增强方法提升复杂口吃检测，解决多模态信息崩溃问题。
StutterFuse: Mitigating Modality Collapse in Stuttering Detection with Jaccard-Weighted Metric Learning and Gated Fusion
- 基于临床案例记忆库的检索增强分类，避免单纯记忆训练数据。
- 在SEP-28k数据集上达到0.65加权F1，显著优于基线模型。
- 适合需要跨语言泛化和复杂口吃识别的研究与临床应用。
口吃检测在多种不流畅现象重叠时会失效。现有参数化模型因训练数据中缺乏特定组合（如‘卡顿’伴随‘延长’），难以区分复杂、同时发生的不流畅。尽管检索增强生成（RAG）已在自然语言处理中革新了模型的知识依赖方式，但在病理性语音处理中仍未被探索。为此，我们提出StutterFuse，首个用于多标签口吃检测的检索增强分类器（RAC）。通过将Conformer编码器与非参数化临床案例记忆库结合，使模型基于参考而非记忆进行分类。我们识别并解决了‘模态坍塌’问题——即简单检索虽提高召回率但降低精确率。通过引入：(1) SetCon，一种基于杰卡德加权的度量学习目标，优化多标签集合相似性；(2) 门控专家混合融合策略，动态协调声学证据与检索上下文。在SEP-28k数据集上，StutterFuse取得0.65的加权F1分数，超越强基线，并展现出卓越的零样本跨语言泛化能力。
原文摘要 · Abstract (English)
Stuttering detection breaks down when disfluencies overlap. Existing parametric models struggle to distinguish complex, simultaneous disfluencies (e.g., a 'block' with a 'prolongation') due to the scarcity of these specific combinations in training data. While Retrieval-Augmented Generation (RAG) has revolutionized NLP by grounding models in external knowledge, this paradigm remains unexplored in pathological speech processing. To bridge this gap, we introduce StutterFuse, the first Retrieval-Augmented Classifier (RAC) for multi-label stuttering detection. By conditioning a Conformer encoder on a non-parametric memory bank of clinical examples, we allow the model to classify by reference rather than memorization. We further identify and solve "Modality Collapse", an "Echo Chamber" effect where naive retrieval boosts recall but degrades precision. We mitigate this using: (1) SetCon, a Jaccard-Weighted Metric Learning objective that optimizes for multi-label set similarity, and (2) a Gated Mixture-of-Experts fusion strategy that dynamically arbitrates between acoustic evidence and retrieved context. On the SEP-28k dataset, StutterFuse achieves a weighted F1-score of 0.65, outperforming strong baselines and demonstrating remarkable zero-shot cross-lingual generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。