有序1D分层标记让生成搜索更高效,无需训练即可实现文本到图像生成。
(1D) Ordered Tokens Enable Efficient Test-Time Search

- 采用粗到细的1D有序标记结构,提升生成过程中的可搜索性。
- 在测试阶段搜索时,性能优于传统2D网格标记,支持无训练生成。
- 适用于需要高效推理和可控生成的研究者与开发者。
自回归生成模型中的分词是关键环节,将原始数据转化为可建模的单元。传统方法使用局部信息(如图像像素区域或文本词元)并按固定顺序生成。本文提出,具有粗到细结构的1D有序标记器比经典2D网格结构更适合测试阶段搜索——因为其中间状态携带语义信息,便于验证器评估,从而实现有效控制。实验表明,基于粗到细标记训练的自回归模型在测试阶段扩展性更强;更重要的是,仅通过有序标记序列的纯测试阶段搜索,配合图像-文本验证器,即可实现无需训练的文本到图像生成。进一步研究发现,不同搜索算法(Best-of-N、束搜索、前瞻搜索)与不同标记结构及验证器的组合影响显著。结果表明,标记结构对推理阶段可扩展性有重要影响,为自回归模型的测试阶段扩展提供了实用指导。
原文摘要 · Abstract (English)
Tokenization is a key component of autoregressive (AR) generative models, converting raw data into more manageable units for modeling. Commonly, tokens describe local information, such as regions of pixels in images or word pieces in text, and AR generation predicts these tokens in a fixed order. A worthwhile question is whether token structures affect the ability to steer the generation through test-time search, where multiple candidate generations are explored and evaluated by a verifier. Using image generation as our testbed, we hypothesize that recent 1D ordered tokenizers with coarse-to-fine structure can be more amenable to search than classical 2D grid structures. This is rooted in the fact that the intermediate states in coarse-to-fine sequences carry semantic meaning that verifiers can reliably evaluate, enabling effective steering during generation. Through controlled experiments, we find that AR models trained on coarse-to-fine ordered tokens exhibit improved test-time scaling behavior compared to grid-based counterparts. Moreover, we demonstrate that, thanks to the ordered structure, pure test-time search over token sequences (i.e., without training an AR model) can perform training-free text-to-image generation when guided by an image-text verifier. Beyond this, we systematically study how classical search algorithms (best-of-N, beam search, lookahead search) interact with different token structures, as well as the role of different verifiers and AR priors. Our results highlight the impact of token structure on inference-time scalability and provide practical guidance for test-time scaling in AR models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。