arXiv:2606.19944cs.CV2026-06

让文字直接画在图上,精准引导模型看对位置。

Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models

论文配图:Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models
图 1 · 摘自论文原文
  • 用可微渲染将文本作为视觉叠加层画在图像上
  • 在VMCBench上7B模型超越更大商用系统
  • 适合需要精准空间推理的多模态任务

多模态大模型在细粒度空间推理中常丢失正确图像区域,因文本查询缺乏显式几何锚点。现有方法或重训练权重或冗长提示,均难以在保持通用能力的前提下准确定位。本文提出Timage,将多模态理解重构为输入层面的对齐问题:将查询以排版形式直接绘制在图像上。该叠加层由受限薛定谔桥(cSB)生成,分两阶段:第一阶段区域搜索,将噪声向与查询对齐的图像区域迁移,同时遵守硬遮挡屏障保护显著前景;第二阶段外观塑造,通过“墨水预算”正则化控制字形大小,确保可读性与视觉平衡。生成的叠加层作为显式注意力信标,引导模型聚焦空间语义。在VMCBench基准上,搭配7B规模主干模型的Timage明显超越更大规模的专有系统及参数调优基线。研究证明,有意输入重构是增强多模态推理的强大、架构无关手段。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) often lose track of the right image regions during fine-grained spatial reasoning, because a textual query rarely carries any explicit geometric anchor into the pixel domain. Prevailing remedies either rewire the model's weights or pad the prompt with verbose instructions, yet neither reliably pins the language to the correct visual coordinates without eroding the backbone's general competence. We introduce Timage, a paradigm that recasts multimodal understanding as an alignment problem solved at the input: the query is drawn, as a typeset overlay, onto the image itself. The placement and appearance of this overlay are produced by a Constrained Schrödinger Bridge (cSB), an entropic optimal-transport sampler that factorizes layout synthesis into two coupled stochastic stages. The first stage, Region Search, transports noise toward query-aligned image zones while obeying a hard occlusion barrier that protects salient foreground content; the second stage, Appearance Shaping, sizes the glyphs through an ``ink-budget'' regularizer so that the rendered text stays legible and visually balanced. The resulting overlay behaves as an explicit attention beacon that channels the model's focus along spatial semantics. On the VMCBench suite, Timage paired with a modest 7B backbone clearly overtakes far larger proprietary systems as well as parameter-tuned baselines. The study positions deliberate input reconstruction as a powerful, architecture-neutral lever for strengthening multimodal reasoning.

多模态空间推理生成式输入视觉定位

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。