解决语音识别中时间戳漂移问题,不遗忘原有性能。
REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing

- 用回放编辑方法修正时间戳,保持非时间相关输出不变。
- 长间隙时间对齐率从38.7%提升至95.0%,错误减少至223毫秒。
- 仅需1.6%参数更新,适合资源受限场景的模型优化。
现代自回归语音识别系统可将时间戳作为解码词元输出,实现无需帧级对齐或推理后处理的时间戳转录。我们发现,长时间无语音段会导致生成时间戳发生漂移:转录内容仍合理,但时间轴与音频偏离。通过构建短间隙与长间隙基准,在15个生成时间戳的语音-语言系统上研究此问题。直接的时间戳修正微调虽改善对齐,但严重损害其他任务表现,暴露遗忘问题。提出REDDIT(基于回放的分布编辑)框架,轻量级两阶段后训练:第一阶段在模型自身回放解码上下文中编辑时间戳目标,同时保持非时间词元的冻结基础分布;第二阶段进行短编辑前缀精炼。通过结合语音活动检测裁剪片段、插入无语音间隙及已知拼接偏移,构建无需人工转录或标注的时间戳校正监督信号。在Whisper-tiny上,使用34.9小时校正音频,仅更新1.6%参数,长间隙mIoU从38.7%升至95.0%,跨域混合间隙平均偏差从2752毫秒降至223毫秒,同时保持中文语音识别词错误率(MER)在41.3%(普通SFT微调为524.2%)。
原文摘要 · Abstract (English)
Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing. We show that these generated timestamps can drift across long non-speech spans: the transcript may remain plausible, but the decoded time axis drifts away from the audio. We study this non-speech-induced timestamp drift with self-built gap and long-gap benchmarks across 15 evaluated timestamp-producing ASR and audio-language systems. Naive timestamp-corrected fine-tuning improves alignment but can severely degrade non-target ASR behavior, exposing a forgetting problem. We propose REDDIT(REplay-based Distribution eDITing), a lightweight two-stage post-training framework that corrects timestamps while avoiding this catastrophic forgetting: it first edits timestamp targets under the model's own replayed decoder context while matching the frozen base distribution on non-timestamp tokens, then applies a short edited-prefix refinement stage. In this framework, we construct correction supervision without human transcripts or human timestamp annotations by combining VAD-trimmed speech spans with inserted non-speech gaps and known concatenation offsets. On Whisper-tiny, 34.9 hours of targeted correction audio used and only 1.6% of model parameters updated, raising long-gap mIoU from 38.7% to 95.0% and reducing mixed-gap out-of-domain AAS from 2752 ms to 223 ms while preserving CV-en MER at 41.3% (versus 524.2% for ordinary SFT decoder tuning).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。