对比七种提示方法,发现适配模型大小的动态提示更有效。
The Future of MLLM Prompting is Adaptive: A Comprehensive Experimental Evaluation of Prompt Engineering Methods for Robust Multimodal Performance
- 按模型参数分层,测试七类提示策略在多任务上的表现。
- 大模型代码生成准确率达96.88%,但复杂推理仍低于60%。
- 简单提示更高效,结构化提示易引发幻觉且耗时长。
多模态大语言模型(MLLMs)通过融合文本、图像和代码等模态,有望实现类人响应。然而其能力发挥依赖于有效的提示工程。本文对13个开源MLLM在24项任务上系统评估了七种提示方法:零样本、单样本、少样本、思维链、类比、生成知识与树状思维。模型按参数量分为小(<4B)、中(4B-10B)、大(>10B)三类。结果显示,大模型在代码生成等结构化任务中表现优异,少样本提示下准确率最高达96.88%;但在复杂推理与抽象理解任务中普遍表现不佳,准确率常低于60%,幻觉率高。结构化提示在小模型中幻觉率可升至75%,大模型响应时间超20秒。单一提示方法无法兼顾所有任务。因此,结合示例引导与选择性结构化推理的自适应策略,是提升鲁棒性、效率与事实准确性的关键。研究为编程辅助、知识检索与多模态理解等场景提供实用提示建议。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are set to transform how machines process and generate human-like responses by integrating diverse modalities such as text, images, and code. Yet, effectively harnessing their capabilities hinges on optimal prompt engineering. We present a comprehensive experimental evaluation of seven prompt engineering methods applied to 13 open-source MLLMs over 24 tasks spanning Reasoning and Compositionality, Multimodal Understanding and Alignment, Complex Code Generation and Execution, and Knowledge Retrieval and Integration. Our approach stratifies models by parameter count into Small (<4B), Medium (4B-10B), and Large (>10B) categories and compares prompting techniques including Zero-Shot, One-Shot, Few-Shot, Chain-of-Thought, Analogical, Generated Knowledge, and Tree-of-Thought. While Large MLLMs excel in structured tasks such as code generation, achieving accuracies up to 96.88% under Few-Shot prompting, all models struggle with complex reasoning and abstract understanding, often yielding accuracies below 60% and high hallucination rates. Structured reasoning prompts frequently increased hallucination up to 75% in small models and led to longer response times (over 20 seconds in Large MLLMs), while simpler prompting methods provided more concise and efficient outputs. No single prompting method uniformly optimises all task types. Instead, adaptive strategies combining example-based guidance with selective structured reasoning are essential to enhance robustness, efficiency, and factual accuracy. Our findings offer practical recommendations for prompt engineering and support more reliable deployment of MLLMs across applications including AI-assisted coding, knowledge retrieval, and multimodal content understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。