让AI生成布局时能看图纠错,自动优化排版质量。
Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement
- 用视觉反馈驱动迭代优化,根据渲染效果自我修正。
- 在多个基准上超越主流模型,提升可读性与美观度。
- 适合需要高质量图文排版的AI设计应用。
多模态大语言模型(MLLM)已能将自然语言描述自动转换为结构化布局。现有方法通常采用纯代码生成范式,通过代码表示布局并由图形引擎渲染成图像,但缺乏对最终视觉结果的感知,难以保证可读性和美学质量。本文提出视觉反馈布局模型(VFLM),一种利用视觉反馈进行迭代优化的自改进框架。VFLM具备自省式生成能力,通过视觉信息反思先前问题,反复生成直至达到满意质量。该能力基于强化学习实现,奖励模型结合光学字符识别(OCR)准确率,仅奖励最终输出。实验表明,VFLM在多个基准上持续优于先进MLLM、现有布局模型及代码基线,验证了视觉反馈对面向设计的MLLM至关重要。代码与数据已开源。
原文摘要 · Abstract (English)
Recent advances in Multimodal Large Language Models (MLLMs) have enabled automated generation of structured layouts from natural language descriptions. Existing methods typically follow a code-only paradigm that generates code to represent layouts, which are then rendered by graphic engines to produce final images. However, they are blind to the rendered visual outcome, making it difficult to guarantee readability and aesthetics. In this paper, we identify visual feedback as a critical factor in layout generation and propose Visual Feedback Layout Model (VFLM), a self-improving framework that leverages visual feedback iterative refinement. VFLM is capable of performing adaptive reflective generation, which leverages visual information to reflect on previous issues and iteratively generates outputs until satisfactory quality is achieved. It is achieved through reinforcement learning with a visually grounded reward model that incorporates OCR accuracy. By rewarding only the final generated outcome, we can effectively stimulate the model's iterative and reflective generative capabilities. Experiments across multiple benchmarks show that VFLM consistently outperforms advanced MLLMs, existing layout models, and code-only baselines, establishing visual feedback as critical for design-oriented MLLMs. Our code and data are available at https://github.com/FolSpark/VFLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。