评测多模态大模型在少样本下的理解能力，发现提示技巧效果有限。
FewMMBench: A Benchmark for Multimodal Few-Shot Learning
- 构建涵盖多种任务的少样本评估基准，聚焦上下文学习与思维链提示。
- 26个开源多模态模型测试显示，指令微调模型少样本提升不明显甚至下降。
- 检索式示例和增加上下文长度带来的性能改进微弱，暴露模型瓶颈。
随着多模态大语言模型（MLLMs）在处理图文交织数据方面不断进步，其少样本学习能力的评估仍面临挑战。本文提出FewMMBench，一个全面的基准，用于在少样本条件下评估MLLMs，重点关注上下文学习（ICL）与思维链（CoT）提示。该基准覆盖从属性识别到时间推理等多样化多模态理解任务，支持跨任务类型、模型族与提示策略的系统性分析。我们在零样本、少样本及CoT增强少样本设置下，评估了来自六个模型家族的26个开源MLLMs。结果表明，指令微调模型在零样本下表现优异，但在添加演示或使用CoT推理时收益甚微，甚至出现性能退化；基于检索的演示与增加上下文长度也仅带来有限提升。这些发现凸显FewMMBench作为诊断与推动多模态大模型少样本能力发展的严格测试平台的价值。数据集已公开：https://huggingface.co/datasets/mustafaa/FewMMBench。
原文摘要 · Abstract (English)
As multimodal large language models (MLLMs) advance in handling interleaved image-text data, assessing their few-shot learning capabilities remains an open challenge. In this paper, we introduce FewMMBench, a comprehensive benchmark designed to evaluate MLLMs under few-shot conditions, with a focus on In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. Covering a diverse suite of multimodal understanding tasks, from attribute recognition to temporal reasoning, FewMMBench enables systematic analysis across task types, model families, and prompting strategies. We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings. Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning. Retrieval-based demonstrations and increased context size also yield limited gains. These results highlight FewMMBench as a rigorous testbed for diagnosing and advancing few-shot capabilities in multimodal LLMs. The data is available at: https://huggingface.co/datasets/mustafaa/FewMMBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。