用视觉化方法让小模型在多模态上表现媲美专业模型
MARVIS: Modality Adaptive Reasoning over VISualizations
- 将数据嵌入空间转为视觉图像，用视觉大模型分析
- 30亿参数模型跨视觉、音频、生物、表格领域表现优秀
- 无需领域训练，性能比Gemini 2.0高16%，适合多场景应用
机器学习预测常依赖小型（小于10亿参数）专用模型，在特定领域表现优异但缺乏灵活性。大语言模型和视觉语言模型虽具通用性，但在非传统模态和长尾领域表现较差。我们提出MARVIS（Modality Adaptive Reasoning over VISualizations），将潜在嵌入空间转换为视觉表示，利用视觉语言模型的空间与细粒度推理能力进行预测。仅用一个30亿参数模型，MARVIS在视觉、音频、生物和表格领域均取得竞争力表现，平均性能超越Gemini 2.0达16%。该方法显著缩小了通用模型与专用模型之间的差距，且无需任何领域特定训练。代码与数据集已开源。
原文摘要 · Abstract (English)
Predictive applications of machine learning often rely on small (sub 1 Bn parameter) specialized models tuned to particular domains or modalities. Such models often achieve excellent performance, but lack flexibility. LLMs and VLMs offer versatility, but typically underperform specialized predictors, especially on non-traditional modalities and long-tail domains. We propose MARVIS (Modality Adaptive Reasoning over VISualizations), a system that transforms latent embedding spaces into visual representations and then leverages the spatial and fine-grained reasoning skills of VLMs to interpret the visualizations and utilize them for predictions successfully. MARVIS achieves competitive performance across vision, audio, biological, and tabular domains using a single 3B parameter model, yielding results that beat Gemini 2.0 by 16% on average. MARVIS drastically reduces the gap between LLM/VLMs approaches and specialized domain-specific methods, without requiring any domain-specific training. Code and datasets are available at https://github.com/penfever/marvis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。