通过可学习任务标记提升多模态模型对视觉任务的精准理解能力
Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
- 引入可微任务偏好机制，用细粒度视觉任务标签训练模型
- 在多任务联合训练中实现性能超越单任务训练，整体提升14.6%
- 零样本表现媲美监督模型，适合需要泛化能力的多任务场景
当前多模态大语言模型虽在多数视觉应用中具备全面感知与推理能力，但在细粒度视觉理解上表现不足。现有方法或依赖工具使用，或统一特定视觉任务至自回归框架，常以牺牲整体多模态性能为代价。为此，我们提出任务偏好优化（TPO），一种基于典型细粒度视觉任务的可微任务偏好机制。TPO引入可学习任务标记，连接多个任务专用头与多模态大语言模型。训练过程中利用丰富的视觉标签，显著增强模型的多模态能力与任务专精表现。通过在TPO中进行多任务协同训练，观察到协同增益，使各任务性能超越单任务训练。以VideoChat和LLaVA为例的实例表明，相较基线模型，整体多模态性能提升14.6%。此外，MLLM-TPO展现出强零样本能力，在多种任务上表现接近最先进的监督模型。代码将于https://github.com/OpenGVLab/TPO发布。
原文摘要 · Abstract (English)
Current multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals although they give comprehensive perception and reasoning in a spectrum of vision applications. Recent studies either develop tool-using or unify specific visual tasks into the autoregressive framework, often at the expense of overall multimodal performance. To address this issue and enhance MLLMs with visual tasks in a scalable fashion, we propose Task Preference Optimization (TPO), a novel method that utilizes differentiable task preferences derived from typical fine-grained visual tasks. TPO introduces learnable task tokens that establish connections between multiple task-specific heads and the MLLM. By leveraging rich visual labels during training, TPO significantly enhances the MLLM's multimodal capabilities and task-specific performance. Through multi-task co-training within TPO, we observe synergistic benefits that elevate individual task performance beyond what is achievable through single-task training methodologies. Our instantiation of this approach with VideoChat and LLaVA demonstrates an overall 14.6% improvement in multimodal performance compared to baseline models. Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across various tasks, performing comparably to state-of-the-art supervised models. The code will be released at https://github.com/OpenGVLab/TPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。