提出新型优化框架，解决多模态模型训练中的模态竞争问题。
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
- 采用二级优化与多级方差校正，缓解跨模态梯度不一致
- 在8192大批次下稳定训练，生成与理解性能同步提升
- 适合大规模多模态基础模型训练，显著加速收敛
自回归下一个词训练为图像生成和文本理解提供了统一范式，但也引发强烈的模态竞争，导致优化不稳定并限制大批次扩展。我们发现一阶优化器如AdamW对跨模态梯度异质性敏感，而二阶预条件化（尤其是SOAP）能提供更稳定的多模态对齐基础。基于此，我们提出 extit{ML-FOP-SOAP}，一种具有多级方差校正的二阶优化框架。通过费舍尔正交投影抑制方差引起的模态冲突，降低视觉生成与文本理解间的权衡。为在大梯度累积下实用，引入分层折叠策略，在低微步开销下捕捉细粒度方差。在Janus和Emu3上的实验显示，双模态性能持续提升，且在批量大小8192时训练稳定。相比AdamW，本方法样本效率最高提升1.4倍，墙钟时间加速达1.5倍，为扩展多模态基础模型提供稳健优化器。
原文摘要 · Abstract (English)
Autoregressive next-token training offers a unified formulation for image generation and text understanding, but it also creates strong modality competition that destabilizes optimization and limits large-batch scaling. We show that first-order optimizers such as AdamW are vulnerable to cross-modality gradient heterogeneity, while second-order preconditioning, particularly SOAP, provides a more stable basis for multimodal alignment. Building on this insight, we propose \emph{ML-FOP-SOAP}, a second-order optimization framework with Multi-Level Variance Correction. Our Fisher-Orthogonal Projection suppresses variance-induced modality conflicts, reducing the trade-off between visual generation and textual understanding. To make this practical under large gradient accumulation, we introduce a hierarchical folding strategy that captures fine-grained variance with low micro-step overhead. Experiments on Janus and Emu3 show consistent gains across both modalities and stable training at batch size 8192. Compared with AdamW, our method improves sample efficiency by up to $1.4\times$ and accelerates wall-clock training by up to $1.5\times$, offering a robust optimizer for scaling multimodal foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。