不训练模型，用深度分层提示提升多模态大模型的推理能力。
ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Way
- 通过单目深度估计将场景分层，生成分区域的深度感知描述。
- 在POPE和GQA测试中，多个模型的幻觉率下降，推理准确率提升。
- 无需修改模型，适配黑箱多模态大模型，部署轻便高效。
我们提出ByDeWay，一种无需训练的框架，用于提升多模态大语言模型（MLLMs）性能。该方法采用名为分层深度提示（LDP）的新颖提示策略，通过单目深度估计将场景分为近、中、远三层，利用具备定位能力的视觉-语言模型生成区域特定的描述。这些结构化的深度感知描述被附加至图像-问题提示中，增强空间上下文信息，引导MLLM生成更准确、更少幻觉的回答。该方法轻量、模块化，兼容黑箱MLLM。在对幻觉敏感（POPE）和推理密集型（GQA）基准上的实验表明，多种MLLM均取得一致性能提升，验证了零训练环境下深度感知提示的有效性。
原文摘要 · Abstract (English)
We introduce ByDeWay, a training-free framework designed to enhance the performance of Multimodal Large Language Models (MLLMs). ByDeWay uses a novel prompting strategy called Layered-Depth-Based Prompting (LDP), which improves spatial reasoning and grounding without modifying any model parameters. It segments the scene into closest, mid-range, and farthest layers using monocular depth estimation, then generates region-specific captions with a grounded vision-language model. These structured, depth-aware captions are appended to the image-question prompt, enriching it with spatial context. This guides MLLMs to produce more grounded and less hallucinated responses. Our method is lightweight, modular, and compatible with black-box MLLMs. Experiments on hallucination-sensitive (POPE) and reasoning-intensive (GQA) benchmarks show consistent improvements across multiple MLLMs, validating the effectiveness of depth-aware prompting in a zero-training setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。