让视觉编码器理解文字语义,提升图文压缩效果
Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression

- 用自监督方式对齐文本与图像路径表示
- 在VTCBench上将模型得分从35.10提升至54.02
- 适合关注图文压缩与跨模态对齐的研究者
视觉-文本压缩(VTC)将长文本渲染为图像,并通过视觉编码器(ViT)编码,将数千个文本标记压缩为更少的视觉标记。然而,由于ViT主要在自然图像上预训练,其捕捉的是视觉属性(如字形、字体大小、布局),而非语言语义,导致渲染后图像的表示与原生文本表示产生偏差。我们称这种跨路径不一致为关键但被忽视的瓶颈。提出SPIRAL(自提升路径整合与重对齐)框架,仅利用模型自身文本路径行为作为监督信号,无需外部教师或额外标注。SPIRAL在两个互补粒度上运作:基于策略的局部忠实性蒸馏(OPD),以及基于偏好优化的全局一致性(DPO)。在VTCBench上,SPIRAL将Qwen3-VL-8B的综合得分从35.10提升至54.02,接近原生文本输入性能(55.60),并超越最大达30倍的模型。两种机制互补:OPD在检索任务中表现优异且样本高效,而DPO在推理和记忆任务中更强,数据规模扩展性更好。SPIRAL的优势还泛化到域外基准,证实有效VTC依赖于将渲染图像表示重新对齐至原生文本语义。
原文摘要 · Abstract (English)
Vision-Text Compression (VTC) renders long texts into images and encodes them through the vision encoder (ViT), compressing thousands of text tokens into far fewer visual tokens. However, since the ViT is pretrained predominantly on natural images, it captures visual attributes (glyphs, font sizes, layout) rather than linguistic semantics, causing rendered-image representations to diverge from native-text representations. We term this cross-path inconsistency and show, via rendering perturbation experiments, that it is a critical yet overlooked bottleneck of VTC. We propose SPIRAL (Self-improving Path Integration and Realignment), a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision, requiring no external teachers or additional annotations. SPIRAL operates at two complementary granularities: token-level on-policy distillation (OPD) for local faithfulness, and sequence-level preference optimization (DPO) for global coherence. On VTCBench, SPIRAL improves the overall score of Qwen3-VL-8B from 35.10 to 54.02, approaching the native text-input performance (55.60) and outperforming models up to 30x larger. The two granularities exhibit complementary strengths: OPD excels at retrieval and is sample-efficient, while DPO is stronger on reasoning and memory and scales better with data. SPIRAL's benefits also generalize to out-of-domain benchmarks, confirming that effective VTC hinges on aligning rendered-image representations back to native-text semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。