arXiv:2602.10230cs.LGcs.SD2026-02

用音频模型内部特征直接定位时间,速度提升50倍以上

Encode Once, Decode Never: Reusing Audio LM Internals for Efficient Temporal Localization

  • 不生成文本令牌,直接复用模型内部帧级特征进行定位
  • 推理速度超传统方法50倍,长音频上仍保持高精度
  • 适合需要实时定位的场景,如语音分析、会议转录

音频语言模型将输入音频转换为丰富的帧级表示,但传统的时序定位方法将时间戳生成为文本令牌序列,放弃了帧级表示而采用自回归解码。这种方式速度慢且无法并行,且在训练分布外的时间长度上容易产生幻觉。本文提出内部帧级复用方法,训练音频语言模型直接利用自身内部音频表示进行时序定位,完全跳过令牌生成过程。引入轻量级预测头,通过二分类帧判别和一种新型非齐次泊松过程(IHP)损失来建模事件发生强度。在词定位、说话人辨识和事件定位任务中,该方法实现超过50倍的推理加速,在超出训练分布的音频时长上表现出鲁棒性,而传统令牌生成模型则完全失效。复用帧级表示可达到与微调后的令牌基线相当甚至更优的定位精度。

原文摘要 · Abstract (English)

Audio language models process input audio into rich frame-level representations, but the standard approach to temporal localization generates timestamps as sequences of text tokens, which discards the frame-level representations in favor of autoregressive decoding. However, generating timestamps as tokens is slow and not parallelizable, and tends to hallucinate when producing timestamps outside the training distribution. We propose internal frame-level reuse, a method that trains audio LMs to reuse their own internal audio representations for temporal localization directly, bypassing token generation altogether. We introduce a lightweight prediction head trained via different frame-level objectives: a binary frame classifier and a novel inhomogeneous Poisson process (IHP) loss that models temporal event intensity. Across word localization, speaker diarization, and event localization tasks, our approach can achieve a >50x inference speedup over token-based generation and demonstrates robust length generalization, maintaining high accuracy on out-of-distribution audio durations where token-based models collapse completely. We find that reusing audio frame-level representations yields localization accuracies comparable to (and often better than) finetuned token-based baselines.

音频定位效率优化帧级表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。