arXiv:2505.19554cs.CV2025-05

用大模型+图结构融合生成更合理的人机布局,支持人工逐步优化。

Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation

  • 将图网络与大语言模型结合,用图结构作为层级先验知识
  • 在RICO数据集上mIoU达0.68,布局合理性显著提升
  • 中间图矩阵可人工编辑,适合需要人机协作的设计场景

人工布局设计耗时且复杂,自动化布局生成对多设备应用至关重要。现有基于图的方法生成能力有限,常产生不合理布局;视觉生成模型则易忽略原始结构信息,导致组件重叠。为此,我们提出聚合结构表示(ASR)模块,将图网络与大语言模型(LLM)融合,在保留结构信息的同时增强生成能力。该新流程以图特征作为层级先验知识,替代多模态大语言模型(MLLM)中的传统视觉变压器(ViT)模块,首次实现对完整布局的预测。此外,用于输入LLM的中间图矩阵可人工编辑,支持渐进式、以人为本的设计生成。在RICO数据集上的综合评估显示,ASR在定量指标(均交并比mIoU)和定性用户研究中表现优异。通过关系特征采样,还实现了多样化的布局生成,进一步提升了方法的适应性与创造性。

原文摘要 · Abstract (English)

Time consumption and the complexity of manual layout design make automated layout generation a critical task, especially for multiple applications across different mobile devices. Existing graph-based layout generation approaches suffer from limited generative capability, often resulting in unreasonable and incompatible outputs. Meanwhile, vision based generative models tend to overlook the original structural information, leading to component intersections and overlaps. To address these challenges, we propose an Aggregation Structural Representation (ASR) module that integrates graph networks with large language models (LLMs) to preserve structural information while enhancing generative capability. This novel pipeline utilizes graph features as hierarchical prior knowledge, replacing the traditional Vision Transformer (ViT) module in multimodal large language models (MLLM) to predict full layout information for the first time. Moreover, the intermediate graph matrix used as input for the LLM is human editable, enabling progressive, human centric design generation. A comprehensive evaluation on the RICO dataset demonstrates the strong performance of ASR, both quantitatively using mean Intersection over Union (mIoU), and qualitatively through a crowdsourced user study. Additionally, sampling on relational features ensures diverse layout generation, further enhancing the adaptability and creativity of the proposed approach.

布局生成大语言模型图神经网络人机协同

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。