arXiv:2411.17673cs.CV2024-11CVPR被引 41

用对话生成草图,无需训练即可实时互动作画。

SketchAgent: Language-Driven Sequential Sketch Generation

  • 通过上下文示例引导大模型逐笔生成草图
  • 支持自然语言指令与草图的动态交互
  • 适合创意设计、人机协作等场景

草图是表达想法、快速探索和视觉沟通的通用工具,广泛应用于多个领域。尽管人工智能在内容生成和人机交互方面取得显著进展,但捕捉人类草图的动态性和抽象性仍具挑战。本文提出SketchAgent,一种无需训练或微调的语言驱动序列草图生成方法,允许用户通过动态对话式交互创建、修改和优化草图。该方法利用现成多模态大模型的序列特性与丰富先验知识,引入一种直观的草图语言,通过上下文示例注入模型,使其以字符串动作“绘制”。这些动作被转化为矢量图形并渲染至像素画布,可反复访问用于后续任务。通过逐笔作画,该代理捕捉了草图固有的演化与动态特征。实验表明,SketchAgent能从多样化提示生成草图,支持对话驱动作画,并与人类用户实现有意义的合作。

原文摘要 · Abstract (English)

Sketching serves as a versatile tool for externalizing ideas, enabling rapid exploration and visual communication that spans various disciplines. While artificial systems have driven substantial advances in content creation and human-computer interaction, capturing the dynamic and abstract nature of human sketching remains challenging. In this work, we introduce SketchAgent, a language-driven, sequential sketch generation method that enables users to create, modify, and refine sketches through dynamic, conversational interactions. Our approach requires no training or fine-tuning. Instead, we leverage the sequential nature and rich prior knowledge of off-the-shelf multimodal large language models (LLMs). We present an intuitive sketching language, introduced to the model through in-context examples, enabling it to "draw" using string-based actions. These are processed into vector graphics and then rendered to create a sketch on a pixel canvas, which can be accessed again for further tasks. By drawing stroke by stroke, our agent captures the evolving, dynamic qualities intrinsic to sketching. We demonstrate that SketchAgent can generate sketches from diverse prompts, engage in dialogue-driven drawing, and collaborate meaningfully with human users.

草图生成大模型人机交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。