用时间提示+强化学习，让语音模型更准地识别事件起止时间。
Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt
- 将时间戳转为嵌入，插在音频特征中作为时间坐标提示。
- 在监督微调后用强化学习优化，提升事件定位精度。
- 适合需要精确时间感知的音频分析任务，如事件检测、密集字幕。
大型音频-语言模型（LALMs）实现了通用音频理解，在多种音频任务中表现优异。然而，这些模型在时间感知方面仍存在挑战（例如推断事件的开始与结束时间），限制了其在细粒度场景中的应用。为此，我们提出音频侧时间提示（Audio-Side Time Prompt），并采用强化学习（RL）构建TimePro-RL框架以实现细粒度时间感知。具体而言，我们将时间戳编码为嵌入，并将其插入音频特征序列中作为时间坐标，引导模型学习。此外，在监督微调（SFT）后引入强化学习，直接优化时间对齐性能。实验表明，TimePro-RL在多个音频时间任务中均取得显著提升，包括音频定位、声音事件检测和密集音频字幕生成，验证了其鲁棒有效性。
原文摘要 · Abstract (English)
Large Audio-Language Models (LALMs) enable general audio understanding and demonstrate remarkable performance across various audio tasks. However, these models still face challenges in temporal perception (e.g., inferring event onset and offset), leading to limited utility in fine-grained scenarios. To address this issue, we propose Audio-Side Time Prompt and leverage Reinforcement Learning (RL) to develop the TimePro-RL framework for fine-grained temporal perception. Specifically, we encode timestamps as embeddings and interleave them within the audio feature sequence as temporal coordinates to prompt the model. Furthermore, we introduce RL following Supervised Fine-Tuning (SFT) to directly optimize temporal alignment performance. Experiments demonstrate that TimePro-RL achieves significant performance gains across a range of audio temporal tasks, such as audio grounding, sound event detection, and dense audio captioning, validating its robust effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。