通过共享骨干并行扩展多模态大模型计算，提升性能且灵活分配视觉语言资源。
ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs
- 用共享骨干+分支前缀参数实现多路并行计算，避免重复参数
- 在130亿词上端到端训练，不同任务最优视觉语言计算配比不同
- 适合需要灵活优化视觉与语言计算比例的多模态应用
现有多模态大模型的扩展策略通常只增加模型参数或序列推理计算，导致显著的内存或延迟开销。更重要的是，大多数方法无法改变视觉变换器（ViT）与大语言模型（LLM）之间固定的计算分配，限制了针对任务的优化。为此，我们提出并行视觉-语言（ParVL）扩展框架，通过复用现有ViT和LLM主干参数，在多个视觉与语言分支间实现并行计算。该框架的核心问题是：在固定主干参数预算下，如何在视觉与语言模态间分配额外的共享计算？我们为每条并行计算流使用分支特定的前缀参数，并在约130亿个标记上通过全参数监督微调进行端到端训练。系统研究了ViT编码器与LLM解码器之间的计算分配权衡。实验表明，相比同配置单分支基线，ParVL在整体多模态性能上取得提升，且最佳视觉-语言计算分配随任务变化而异。代码已开源。
原文摘要 · Abstract (English)
Existing scaling strategies for Multimodal Large Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the Vision Transformer and the Large Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel Vision-Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the vision and language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall multimodal performance over same-recipe single-branch baselines, and the best evaluated vision--language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。