不重训练模型，用残差适配器提升文本渲染清晰度。
Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering
- 通过残差分支和共享分布码本，后置优化视觉分词器。
- 在TextAtlas上将OCR准确率从12.75%提升至36.81%。
- 兼容已有AR模型，适合需要精细文本生成的场景。
视觉自回归（AR）模型通过预测离散令牌生成图像，由视觉分词器解码。尽管整体图像生成能力强劲，但在文本渲染方面仍表现不佳，常出现模糊笔画和字母形状扭曲。本文将此问题归因于分词器难以重建细粒度细节。直接改进分词器虽可行，但需同时重训分词器与AR模型，成本高昂。为此，我们提出残差解码器适配器（RDA），在不改变原有令牌空间的前提下，后置升级现有分词器。具体而言，引入两个新组件：(i) 与原码本共享令牌分布的配对码本；(ii) 并行分支，在像素空间学习重构图像与真实图像之间的微小差异（残差）。该设计实现非侵入式增强，同时保持与原有AR模型的兼容性。RDA显著提升文本渲染效果，在TextAtlas基准测试中，微调后的Janus-Pro OCR准确率从24.52%升至58.26%（TextVisionBlend），从12.75%升至36.81%（StyledTextSynth）。代码已开源：https://github.com/CSU-JPG/RDA。
原文摘要 · Abstract (English)
Visual Autoregressive (AR) models generate images by predicting discrete tokens that are decoded by a visual tokenizer. Despite demonstrating strong overall image generation ability, they still underperform on text rendering with blur strokes and disrupt letter shapes. In this work, we trace this limitation to the visual tokenizer, which struggles to reconstruct fine-grained detail. Improving the tokenizer is straightforward but expensive, as it necessitates retraining both the tokenizer and the AR model. Can we improve text rendering performance of AR models without retraining the existing tokenizer and AR model? To achieve this, we propose the Residual Decoder Adapter(RDA) that upgrades an existing tokenizer post-hoc without changing its token space. Specifically, it refines the decoder output of the visual tokenizer by introducing two novel components: (i) a paired codebook that shares the token distribution with the original one; (ii) a parallel branch to learn the tiny differences (residual) between the reconstructed image and the ground-truth images in the pixel space. This residual design allows us to enhance the tokenizer non-invasively while preserving compatibility with prior AR models. RDA substantially improves text rendering significantly by a large margin. For instance, we boost finetuned Janus-Pro OCR accuracy rises from 24.52% to 58.26% (TextVisionBlend), from 12.75% to 36.81% (StyledTextSynth) on competitive TextAtlas benchmark. The code is available at https://github.com/CSU-JPG/RDA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。