arXiv:2608.29999cs.SDcs.AI2026-08中稿 · EMNLP

首个统一处理音频、语音、音乐时间定位的模型,让大音视频模型能精准标注事件发生时刻。

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

论文配图:TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
图 1 · 摘自论文原文
  • 用原子时间标记符+时间感知投影器,让模型理解音频帧的精确时间位置。
  • 在10K样本测试集上超越现有顶尖模型,尤其在语音和音乐时间定位任务中显著领先。
  • 首次将强化学习用于统一时间定位,以可验证的时间奖励直接优化评估指标。

大型音视频模型(LALMs)虽能描述音频片段内容,却无法为识别出的事件、说话人或声音分配准确时间戳,这限制了其在语音识别和密集音频描述等下游任务中的应用。本文提出TEMPO(时序对齐多任务后训练),首个统一处理音频、语音与音乐时间定位的模型。核心创新包括:原子时间标记符、注入正弦时钟编码的时间感知投影器,以及距离感知的高斯损失函数。训练采用从合成到真实数据的渐进式课程策略。我们首次将强化学习引入统一时间定位任务,使用GRPO并设计可验证的时间奖励机制,直接优化评估目标。尽管强化学习阶段带来小幅提升,主要性能增益来自监督微调(SFT)。为此,我们构建了含11.9万样本的训练集和含1万样本的评测基准,涵盖五个任务。在该基准上,TEMPO优于Audio Flamingo Next和Qwen3-Omni,这两者均为专门训练于带时间戳数据的先进模型。实验表明,SFT贡献了大部分性能提升,而GRPO起到稳定且适度的优化作用。

原文摘要 · Abstract (English)

Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present TEMPO (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech, and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, TEMPO outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements.

音频理解时间定位多模态强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。