统一多模态模型中，理解与生成能力在对齐时相互冲突，生成质量难提升。
Do Understanding and Generation Fight? A Diagnostic Study of DPO for Unified Multimodal Models
- 用DPO对齐理解与生成，但生成质量未改善
- 7B模型上生成CLIPScore无提升（Δ<0.2，p>0.5）
- VQ离散化是关键瓶颈，适合研究统一模型的开发者
统一多模态模型使用共享语言模型骨干同时处理理解与生成任务。能否通过直接偏好优化（DPO）同步对齐两种能力？我们首次系统性研究该问题，将DPO应用于1B和7B参数的Janus-Pro模型，采用七种训练策略和两种后处理方法。核心发现为负面：在所有测试条件下，生成质量均难以通过DPO对齐。7B模型上，所有方法均未提升生成CLIPScore（|Δ| < 0.2，p > 0.5，n=200/种子，3个种子）；1B模型上，所有方法均导致生成性能下降，且结果在真实-生成与模型-模型偏好数据类型、150–288对数据量下均成立。梯度分析显示：理解与生成梯度近乎正交（cos ≈ 0），且受VQ token数量不对称（576个生成令牌 vs. ~30–100个文本令牌）驱动，存在约11–14倍的幅度失衡。此失衡是多任务DPO中的主导干扰机制。幅度平衡虽使理解指标略有正向提升（+0.01–0.04 VQA，但不显著），但生成差距持续存在。我们指出离散VQ编码是潜在结构瓶颈——由生成DPO损失收敛至ln(2)支持，并为基于VQ的统一模型实践者提供实用建议。
原文摘要 · Abstract (English)
Unified multimodal models share a language model backbone for both understanding and generating images. Can DPO align both capabilities simultaneously? We present the first systematic study of this question, applying DPO to Janus-Pro at 1B and 7B parameters under seven training strategies and two post-hoc methods. The central finding is negative: generation quality resists DPO alignment across all tested conditions on this architecture. No method improves generation CLIPScore at 7B (|Delta| < 0.2, p > 0.5 at n=200 per seed, 3 seeds); at 1B, all methods degrade generation, and the result holds across preference data types (real-vs-generated and model-vs-model) and the data volumes tested (150-288 pairs). Gradient analysis reveals why: understanding and generation gradients are near-orthogonal (cos ~ 0) with ~11-14x magnitude imbalance driven by VQ token count asymmetry (576 generation tokens vs. ~30-100 text tokens). This imbalance is the dominant interference mechanism in multi-task DPO; magnitude-balancing yields directionally positive understanding deltas (+0.01-0.04 VQA, though individually not significant), but the generation gap persists regardless. We identify discrete VQ tokenization as a likely structural bottleneck -- supported by the generation DPO loss converging to ln(2) -- and provide practical guidance for practitioners working with VQ-based unified models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。