让大模型同时学会文本、图像和音频,还能不丢掉各自优势。
OPOD: On-Policy Omni Distillation

- 按输出内容自动选对应模态老师指导,只在老师更支持时才干预。
- 在12个基准上均超越现有方法,30B模型全胜且领先1.7分以上。
- 适合需要多模态统一能力的落地场景,如智能助手、跨模态搜索。
全能模型统一处理文本、图像和音频,但联合提升难度大,因对混合多模态数据微调常会削弱各模态教师的优势。近期流行的在线策略蒸馏(OPD)通过采样当前学生生成的响应,比较教师与学生在该路径上的下一个词分布,实现密集监督并降低训练与推理间的差异。然而,标准OPD难以扩展至多个模态教师,其指导可能引发冲突,且强制匹配每个教师分布会限制学生突破教师范围。为此,我们提出在线策略全能蒸馏(OPOD),将文本、图像和音频教师整合为一个统一模型。OPOD根据响应内容路由至对应教师,独立控制各教师,并仅当教师对生成词赋予更高概率时施加引导。选定教师还评估答案置信度及推理是否增强答案支持。在十二个基准上的实验表明,OPOD在三种模型规模下均取得最佳平均性能,分别达到70.8、51.7和46.2,优于最强对比方法2.1、1.8和1.7分。30B模型在所有十二个基准上超过基础模型和融合强化学习训练,十一项排名第一或第二,即使包含教师模型也表现优异。部署时仅保留学生模型。
原文摘要 · Abstract (English)
Omni-modal models provide a unified interface for text, images, and audio. However, improving these abilities together remains difficult, as post-training on pooled multimodal data often fails to preserve the strengths of modality teachers. On-policy distillation (OPD) has recently become popular in model post-training. It samples responses from the current student and compares the teacher's and student's next-token distributions along those responses, yielding dense supervision while reducing the mismatch between training and inference. Despite these advantages, standard OPD does not readily extend to several modality teachers. Their guidance may favor conflicting changes to the shared model, while matching each teacher's next-token distribution can prevent the student from moving beyond that teacher. To address these challenges, we propose On-Policy Omni Distillation (OPOD), which consolidates text, image, and audio teachers into one omni model. OPOD routes each response to the corresponding teacher, controls the teachers independently, and applies guidance only when the teacher assigns a higher probability to the generated token. The selected teacher also evaluates answer confidence and whether the reasoning increases support for the answer. Extensive experiments on twelve benchmarks show that OPOD achieves the best average at three model scales, reaching 70.8, 51.7, and 46.2 and outperforming the strongest comparator by 2.1, 1.8, and 1.7 points. At 30B, it surpasses the base model and pooled RL training on all twelve benchmarks, and ranks first or second on eleven even when the teachers are included. Only the student is retained for deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。