用生成任务辅助训练,提升视觉理解能力且不增加推理负担
Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

- 将生成任务作为辅助监督,通过解耦架构让生成损失增强视觉表征
- 在多种模型规模下显著提升感知与空间理解,尤其在空间精度上表现突出
- 训练后丢弃生成分支,零推理开销,适合部署于现有大模型系统
尽管多模态大语言模型取得了显著进展,但视觉理解和生成通常被视为相互分离的目标。现有统一框架常依赖离散视觉标记化或扩散目标,其生成目标与视觉理解模型所使用的连续表示不一致,导致难以直接提升预训练模型。本文提出GAS框架,将视觉生成重新定义为表征学习的辅助监督信号。具体地,GAS在解耦的混合变换器(MoT)架构中采用下一嵌入预测(NEP)作为跨模态生成范式。通过共享底层主干网络和并行的上层结构,生成损失在不干扰理解层的情况下,增强共享视觉路径的空间精度与视觉保留能力。为进一步强化协同效应,我们设计了需要深度认知基础而非简单合成的高相关性生成任务。在不同模型规模和训练阶段下,GAS均提升了整体多模态理解能力,尤其在感知与空间理解方面效果显著。关键的是,训练完成后丢弃生成分支,带来零推理开销。大量受控对比实验与表征分析揭示了生成引导训练为何有效,并证明该方法是实现更强多模态理解的可行路径。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) have achieved remarkable progress, visual understanding and generation are typically treated as divergent objectives. Existing unified frameworks often rely on discrete visual tokenization or diffusion objectives whose generative targets differ from the continuous representations consumed by visual understanding models, making direct transfer to enhance existing pretrained MLLMs non-trivial. In this work, we present GAS, a generation-guided training framework that reinterprets visual generation as auxiliary supervision for representation learning. Concretely, GAS adapts Next Embedding Prediction (NEP) as a cross-modal generation paradigm within a decoupled Mixture-of-Transformers (MoT) architecture. By maintaining a shared lower trunk and parallel upper layers, GAS lets generation losses enrich the shared visual pathway with finer spatial precision and stronger visual retention while shielding the upper understanding layers from direct generation gradients. To maximize this synergy, we further construct highly correlated generation tasks that demand deep cognitive grounding rather than generic synthesis alone. Across model scales and training stages, GAS improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension. Crucially, because the auxiliary generation branch is discarded after training, these gains incur zero inference overhead. Extensive controlled comparisons and representation-level analyses further clarify when and why generation-guided training benefits understanding, and demonstrate the feasibility of generation-guided training as a practical route to stronger multimodal understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。