提出语音转文本生成的可解释方法SPES，精准定位每步预测依据。
SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation
- 基于频谱图扰动，结合输入与已生成内容解释每一步预测
- 在语音识别与翻译任务上验证，解释结果对人可信且准确
- 适合需要理解生成过程的语音模型调试与应用
随着对可解释模型的需求增长，可解释人工智能在语言技术领域的研究迅速发展，特征归因方法成为核心进展。尽管先前NLP研究已在分类和文本任务中探索此类方法，但语音生成与可解释性的交叉研究仍滞后，现有技术未能考虑当前先进模型的自回归特性，也难以提供细粒度、音素层面有意义的解释。本文提出适用于自回归序列生成任务的语音转文本可解释方法SPES（Spectrogram Perturbation for Explainable Speech-to-text Generation），其解释基于输入频谱图及先前生成的词元。在语音识别与语音翻译任务上的大量评估表明，SPES生成的解释对人类而言具有忠实性与合理性。
原文摘要 · Abstract (English)
Spurred by the demand for interpretable models, research on eXplainable AI for language technologies has experienced significant growth, with feature attribution methods emerging as a cornerstone of this progress. While prior work in NLP explored such methods for classification tasks and textual applications, explainability intersecting generation and speech is lagging, with existing techniques failing to account for the autoregressive nature of state-of-the-art models and to provide fine-grained, phonetically meaningful explanations. We address this gap by introducing Spectrogram Perturbation for Explainable Speech-to-text Generation (SPES), a feature attribution technique applicable to sequence generation tasks with autoregressive models. SPES provides explanations for each predicted token based on both the input spectrogram and the previously generated tokens. Extensive evaluation on speech recognition and translation demonstrates that SPES generates explanations that are faithful and plausible to humans.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。