arXiv:2606.01802cs.SDcs.AI2026-06被引 11

统一音频语言模型,支持语音、音乐、环境音的多任务理解与时间定位。

MOSS-Audio Technical Report

论文配图:MOSS-Audio Technical Report
图 1 · 摘自论文原文
  • 采用音频编码器+适配器+大语言模型架构,12.5Hz时序表征驱动文本生成。
  • 在语音、音乐、环境音上实现高精度字幕生成与时间戳语音识别,关键任务指标领先。
  • 适合构建智能语音助手,尤其擅长带时间线索的复杂音频推理任务。

MOSS-Audio 是一个统一的音频-语言模型,用于语音、环境音和音乐的理解,支持音频字幕生成、时序感知问答、带时间戳的转录及音频引导的推理。该模型结合专用音频编码器、模态适配器和大语言模型:编码器生成12.5 Hz的时序表示,适配器将其映射至解码器空间,解码器自回归生成文本。核心设计包括:DeepStack跨层特征注入,使解码器能获取多层编码器的声学信息;时间标记,通过在音频令牌流中插入时间戳提供显式时序提示。数据层面,设计事件保持的音频标注流程,按事件边界分割原始音频,对语音、音乐、通用音频分别标注,并合并为统一预训练字幕,同时保留分支字幕用于构建任务导向的SFT数据。模型在大规模音频-语言数据上预训练,引入时序感知目标以支持时间定位,随后进行多阶段后训练,提升指令遵循与音频引导推理能力。发布4B与8B两个版本,均含Instruct与Thinking配置。MOSS-Audio在通用音频理解、语音字幕、自动语音识别(ASR)及带时间戳的ASR任务上表现优异,有望成为未来语音代理的核心理解基础。

原文摘要 · Abstract (English)

MOSS-Audio is a unified audio-language model for speech, environmental sound, and music understanding, supporting audio captioning, time-aware question answering, timestamped transcription, and audio-grounded reasoning. MOSS-Audio couples a dedicated audio encoder with a modality adapter and a large language model: the encoder produces 12.5 Hz temporal representations, the adapter projects them into the decoder space, and the decoder generates autoregressive text outputs. Two design choices are central to the system: DeepStack cross-layer feature injection, which exposes the decoder to acoustic information from multiple encoder depths, and time markers, which provide explicit temporal cues by inserting timestamp markers into the audio-token stream. At the data level, we design an event-preserving audio annotation pipeline that segments raw audio at coherent event boundaries, applies branch-specific annotation to speech, music, and general audio, and merges the results into unified captions for pretraining. The intermediate branch-specific captions are further retained to support the construction of task-oriented SFT data. The model is pretrained on large-scale audio-language data, with time-aware objectives incorporated to support temporal grounding, and then undergoes multi-stage post-training to enhance instruction following and audio-grounded reasoning. We release 4B and 8B variants in both Instruct and Thinking configurations. MOSS-Audio achieves strong performance across general audio understanding, speech captioning, ASR, and timestamped ASR, positioning it as a promising understanding foundation for future voice agents.

音频理解多模态语音识别大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。