arXiv:2608.18132cs.CLcs.SD2026-08被引 1

仅靠对齐训练,就能让语音语言模型高效适配新模态。

Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models

论文配图:Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
图 1 · 摘自论文原文
  • 用自生成数据训练语音-文本对,仅微调轻量投影层。
  • 在多个评测集上表现超越大量微调的基线模型。
  • 模型可无缝迁移至新大模型,适合快速部署场景。

多模态大语言模型通常依赖跨模态对齐、监督微调和偏好优化的多阶段流程,需大量任务特定标注。然而预训练大模型已具备强大推理与指令遵循能力。我们提出一种无需指令的对齐仅训练音频语言模型(LALM),冻结音频编码器与大语言模型,仅训练轻量级投影层。借鉴AzeroS思路,使用自生成数据构建(语音,回复)配对,由大模型将描述扩展为自由文本响应。在MMAU、MMAR、MMSU和MMAU-Pro等多个评测集上,该方法以更少数据达到或超过复杂后训练基线。冻结大模型使其保持原生指令遵循能力,可跨模型版本无缝迁移。结果表明,仅通过对齐即可构建性能优异的多模态模型,将多模态扩展简化为轻量投影层训练问题,具备跨模态泛化能力,并能快速适应新发布的大模型。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimization. This pipeline assumes that adapting an LLM to a new modality requires extensive task-specific supervision. However, pretrained LLMs already possess strong reasoning and instruction-following abilities. As LLMs evolve rapidly, an important question remains: can we efficiently transfer these capabilities to a new modality with minimal intervention, and is alignment alone sufficient for building a multimodal model? We introduce an Instruction-Free Alignment-Only large audio-language model (LALM) that keeps both the audio encoder and the LLM fully frozen, learning only a lightweight projector. Borrowing insights from AzeroS [1], we train on (audio, response) pairs from Self-Generated Data Construction, where an LLM expands captions into free-form responses without explicit task instructions. Across MMAU, MMAR, MMSU, and MMAU-Pro, our approach matches or surpasses heavily post-trained baselines using substantially less data. By keeping the LLM frozen, our model preserves its native instruction-following competence and can port seamlessly across model generations. Our results suggest that competitive MLLM can emerge from alignment alone, reducing multimodal extension to a lightweight projector-training problem that generalizes across modalities and adapts rapidly to each new LLM release.

多模态语音生成对齐训练大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。