用大模型实现多语言长语音精准对齐,解决传统方法累积误差问题。
LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech
- 将对齐任务转为填槽机制,用特殊时间标记替代自回归预测。
- 在多语言、跨语言和长语音场景下,累计偏差降低69%~78%。
- 支持非自回归推理,速度快且避免生成幻觉,适合实际应用。
强制对齐(FA)旨在为语音中的词或字符预测起止时间戳,但现有方法依赖特定语言且易产生累积时间偏移。语音大模型(SLLMs)具备多语言理解与长序列处理能力,有望提升多语言、跨语言及长语音场景下的对齐性能。然而,直接沿用SLLM的下一个词预测范式会导致生成幻觉且推理缓慢。为此,我们提出LLM-ForcedAligner,将对齐重构为填槽任务:将时间戳视为离散索引,在文本中插入特殊时间标记作为槽位;基于语音嵌入与带槽文本,模型直接预测各槽位的时间索引。训练时采用因果注意力掩码,输入与标签序列无偏移,仅在槽位位置计算损失,确保每个槽基于自身及前文上下文独立预测。动态槽插入支持任意位置对齐。此外,支持非自回归推理,避免幻觉并提升速度。在多语言、跨语言及长语音场景的实验表明,该方法相较先前方法实现了69%~78%的累计平均偏移相对减少。检查点与推理代码已开源于https://github.com/QwenLM/Qwen3-ASR。
原文摘要 · Abstract (English)
Forced alignment (FA) predicts start and end timestamps for words or characters in speech, but existing methods are language-specific and prone to cumulative temporal shifts. The multilingual speech understanding and long-sequence processing abilities of speech large language models (SLLMs) make them promising for FA in multilingual, crosslingual, and long-form speech settings. However, directly applying the next-token prediction paradigm of SLLMs to FA results in hallucinations and slow inference. To bridge the gap, we propose LLM-ForcedAligner, reformulating FA as a slot-filling paradigm: timestamps are treated as discrete indices, and special timestamp tokens are inserted as slots into the transcript. Conditioned on the speech embeddings and the transcript with slots, the SLLM directly predicts the time indices at slots. During training, causal attention masking with non-shifted input and label sequences allows each slot to predict its own timestamp index based on itself and preceding context, with loss computed only at slot positions. Dynamic slot insertion enables FA at arbitrary positions. Moreover, non-autoregressive inference is supported, avoiding hallucinations and improving speed. Experiments across multilingual, crosslingual, and long-form speech scenarios show that LLM-ForcedAligner achieves a 69%~78% relative reduction in accumulated averaging shift compared with prior methods. Checkpoint and inference code are available at https://github.com/QwenLM/Qwen3-ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。