不训练即可减少Whisper音频令牌量，提升效率。
Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper
- 通过每隔k个令牌保留一个，直接压缩音频令牌序列。
- 降维75%后，多数语音识别任务错误率仅小幅上升。
- 适合追求低延迟的语音模型部署场景。
Whisper 以固定的1500个令牌编码器接口呈现语音，现已成为自动语音识别解码器和基于Whisper的语音语言模型（SpeechLM）的标准表示方式，但其冗余性尚未被充分研究。本文提出步长-k下采样（stride-k subsampling），一种确定性索引操作，在卷积茎干或编码器变换器之后保留每第k个令牌。在五个Whisper规模中，当k=2时，两个位置均能保持基线词错误率（WER），且使用中心核对齐（CKA）分析表明，该稳定性源于茎干处的声学重叠以及编码器输出处注意力机制引起的分布再分配。在两个位置同时应用步长-2，可将音频令牌减少75%，总计算量（GFLOPs）降低52%-58%，多数ASR基准测试错误率略有上升，较难任务上上升更明显。该配置同样适用于三种Whisper-based SpeechLM，强基线模型准确率下降较小，弱基线下降较大，同时端到端延迟降低19.6%-27.4%。该方法无需训练或辅助计算，利用了Whisper预处理阶段的冗余性，表明其音频令牌接口的容量远超下游任务所需。
原文摘要 · Abstract (English)
Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。