让提示变像素，测试模型跨通道理解能力
When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning
- 将问题嵌入图像，测试模型在视觉通道的理解能力
- 跨6个模型4个数据集，平均准确率下降17.8点
- 提出区域对齐方法，不依赖OCR也能提升推理精度
多模态大语言模型越来越多地需要对截图和文档进行推理，而任务描述本身可能以像素形式存在。然而，现有基准通常将问题以文本形式呈现，难以判断模型在不同输入通道上的表现是否一致。我们提出可视化任务语义（VTS），通过将问题移入图像并保持原任务与答案不变，实现可控干预。在6个多模态大模型和4个基准上，所有24个模型-任务组合的准确率均下降，平均降幅达17.8点。模型虽能正确识别图像中的问题文本，却无法有效利用，暴露出超越光学字符识别（OCR）的语义通道差距。为此，我们提出提示区域对齐（prompt-region grounding），其核心是将问题区域与文本语义对齐，并从掩码视图中恢复清晰表示。在相同训练成本下，该方法使四个基准上的VTS准确率从58.0提升至66.3，同时保持原有界面性能，且推理阶段无需OCR或区域元数据。阅读承载任务的文本与将其作为推理指令进行定位，是两种独立能力。
原文摘要 · Abstract (English)
Multimodal large language models increasingly reason over screenshots and documents where the task itself may be written in pixels. Yet benchmarks usually place questions in text, leaving it unclear whether models use the same instruction equally well across channels. We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed. Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average. Models often transcribe the visual question correctly yet fail to use it, exposing a semantic channel gap beyond OCR. To reduce this gap, we present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view. At matched training cost, our method raises four-benchmark VTS accuracy from 58.0 to 66.3 while preserving accuracy on the original interface, and requires no OCR or region metadata at inference. Reading task-bearing text and grounding it as an instruction for reasoning are distinct capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。