剖析Whisper模型在多语言医疗语音识别中的内部变化,揭示训练策略如何影响性能。
Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis

- 通过逐层分析编码器,研究不同微调策略对模型内部表示的影响。
- 直接联合训练使英德混合任务的综合词错误率降至26.30%,优于分阶段训练。
- 模型在适应后仍能保留语言和领域信息,但纠错线索随性能提升而减弱。
医疗自动语音识别(MedASR)需应对专业术语、标注数据稀缺及多语言场景。尽管Whisper等大模型具备强泛化能力,其在医疗与多语言适应后的行为仍缺乏深入理解。本文通过层级编码器分析,探究多语言医疗适配如何重塑Whisper模型的内部表示。比较零样本解码、仅英语微调、仅德语诊断微调、两阶段英文→英德连续训练以及直接英德联合微调在不同规模Whisper模型上的表现。微调显著提升性能,最佳模型取决于适配设置:Whisper-Medium在直接英德联合训练下取得最低英德综合词错误率(26.30%),且英文错误率最低(7.72%);而德语单语微调的Whisper-Large-v3在86个单说话人训练语句的内部诊断测试中达最低德语错误率(44.96%),但未体现稳健泛化能力。层分析显示,两阶段Whisper-Small路径中,英语医疗微调主导编码器变化,多语言继续训练则基本保持已适配表征空间。域与语言信息在各层级均高度可恢复,但线性可恢复的错误预测信号随词错误率降低而减弱。
原文摘要 · Abstract (English)
Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong generalisation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN->EN+DE continuation, and direct EN+DE fine-tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper-Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two-stage Whisper-Small trajectory shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。