用普通大模型做精细图像预测，无需额外模块。
DenseMLLM: Standard Multimodal LLMs for Dense Prediction
- 用新视觉标记监督策略让通用模型处理多任务密集预测。
- 在多个密集预测任务上达到顶尖水平，超越专用模型。
- 适合想用统一框架解决多种视觉任务的研究者。
多模态大语言模型（MLLM）在高层次视觉理解方面表现出色，但将其扩展到语义分割、深度估计等细粒度密集预测任务时，通常需要引入复杂的任务专用解码器和其他定制化设计。这种架构碎片化增加了模型复杂性，偏离了MLLM的通用设计理念，限制了实际应用。本文提出DenseMLLM，基于标准架构，通过一种针对多标签和多任务的新视觉标记监督策略，使标准MLLM无需额外解码器即可完成密集预测。尽管设计极简，该模型在广泛覆盖的密集预测与视觉-语言基准测试中表现卓越，证明标准通用型MLLM可有效支持密集感知而无需架构特化。项目代码已开源：github.com/Eli-YiLi/DenseMLLM。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in high-level visual understanding. However, extending these models to fine-grained dense prediction tasks, such as semantic segmentation and depth estimation, typically necessitates the incorporation of complex, task-specific decoders and other customizations. This architectural fragmentation increases model complexity and deviates from the generalist design of MLLMs, ultimately limiting their practicality. In this work, we challenge this paradigm by accommodating standard MLLMs to perform dense predictions without requiring additional task-specific decoders. The proposed model is called DenseMLLM, grounded in the standard architecture with a novel vision token supervision strategy for multiple labels and tasks. Despite its minimalist design, our model achieves highly competitive performance across a wide range of dense prediction and vision-language benchmarks, demonstrating that a standard, general-purpose MLLM can effectively support dense perception without architectural specialization. This project is available at github.com/Eli-YiLi/DenseMLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。