让视觉语言模型用可编辑的草图解释推理过程，提升可信度
SketchVLM: Vision language models can annotate images to explain thoughts and guide users
- 不修改原图，生成可编辑的SVG草图标注来说明答案
- 在7个任务中最高提升28.5%准确率，标注质量提升1.48倍
- 适合需要可视化解释的AI协作场景，如教育与交互设计
人类在回答图像问题时，常通过指认、标记和绘图来解释思路。而当前视觉语言模型（如Gemini-3-Pro和GPT-5）仅输出文字，难以验证。我们提出SketchVLM，一种无需训练、兼容各类模型的框架，可在输入图像上生成非破坏性、可编辑的SVG叠加层，直观展示模型推理过程。在涵盖视觉推理（迷宫导航、小球轨迹预测、物体计数）与绘画任务（部件标注、连线填图、对象轮廓绘制）的七个基准测试中，SketchVLM将视觉推理准确率最高提升28.5个百分点，标注质量相比图像编辑与微调草图基线提高1.48倍，且标注内容更忠实于模型陈述的答案。单轮生成已具备高精度与良好标注质量，多轮交互则进一步拓展人机协作空间。交互演示与代码见https://sketchvlm.github.io/。
原文摘要 · Abstract (English)
When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision-language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model's stated answer. We find that single-turn generation already achieves strong accuracy and annotation quality, and multi-turn generation opens up further opportunities for human-AI collaboration. An interactive demo and code are at https://sketchvlm.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。