arXiv:2605.12242cs.CLcs.AI2026-05ACL被引 1

用指令微调+对比学习,让大模型更懂如何清除多语言语音转录中的口误。

Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs

  • 先标记口误词,再用指令微调大模型重写文本
  • 在印地语、孟加拉语、马拉地语上提升显著,优于强基线
  • 适合语音助手、聊天机器人等需流畅输入的场景

自动语音识别(ASR)转录常包含填充词、重复和起始错误等口误,影响可读性并损害下游应用如聊天机器人和语音助手的可靠性。现有方法多依赖传统模型识别并移除口误词,但常破坏语法结构和语义连贯性。近期研究尝试使用大语言模型(LLMs),但主要聚焦于检测或数据增强,而非完整修正。本文提出多语言修正流程:先用序列标注器标记口误词,再以这些信号指导LLM进行指令微调,生成流畅文本;同时引入对比学习目标,惩罚口误词的重现,强化语法与语义保持。在印地语、孟加拉语和马拉地语上的实验显示,该方法持续优于强基线,包括多语言序列到序列模型。结果表明仅检测不足,结合词级信号、指令微调与对比学习是可行且可扩展的多语言口误修正方案。代码已开源:https://github.com/deepak-kumar-98/Mind-the-Pause。

原文摘要 · Abstract (English)

Automatic Speech Recognition (ASR) transcripts often contain disfluencies, such as fillers, repetitions, and false starts, which reduce readability and hinder downstream applications like chatbots and voice assistants. If left unaddressed, such disfluencies can significantly degrade the reliability of downstream systems. Most existing approaches rely on classical models that focus on identifying disfluent tokens for removal. While this strategy is effective to some extent, it often disrupts grammatical structure and semantic coherence, leading to incomplete or unnatural sentences. Recent literature explored the use of large language models (LLMs); however, these efforts have primarily focused on disfluency detection or data augmentation, rather than performing comprehensive correction. We propose a multilingual correction pipeline where a sequence tagger first marks disfluent tokens, and these signals guide instruction fine-tuning of an LLM to rewrite transcripts into fluent text. To further improve reliability, we add a contrastive learning objective that penalizes the reproduction of disfluent tokens, encouraging the model to preserve grammar and meaning while removing disfluent artifacts. Our experiments across three Indian languages, namely Hindi, Bengali, and Marathi show consistent improvements over strong baselines, including multilingual sequence-to-sequence models. These results highlight that detection-only strategies are insufficient. Combining token-level cues with instruction tuning and contrastive learning provides a practical and scalable solution for multilingual disfluency correction in speech-driven NLP systems. We make the codes publicly available at https://github.com/deepak-kumar-98/Mind-the-Pause.

语音转录大模型多语言口误修正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。