将手绘草图自动转为结构化图表,减少人工干预。
SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches
- 多智能体系统融合草图识别与逻辑推理,逐轮验证生成结果。
- 构建包含6000+样本的Sketch2Diagram基准数据集,覆盖8类图表。
- 适合设计、教育、工程领域快速转化手绘创意为可读图表。
手绘草图是表达和传递想法的自然高效方式。尽管可控自然图像生成取得显著进展,将自由手绘草图转化为结构化机器可读图表仍依赖大量人工,主要挑战源于草图固有的模糊性,缺乏生成所需结构约束与语义精度。为此,我们提出SketchAgent,一个用于自动化将手绘草图转化为结构化图表的多智能体系统。该系统整合草图识别、符号推理与迭代验证,生成语义连贯且结构准确的图表,显著降低人工成本。为评估方法有效性,我们构建Sketch2Diagram基准,涵盖流程图、有向图、模型架构等8类图表,包含超6000个高质量样本,具备词级标注、标准化预处理与严格质量控制。通过简化图表生成流程,SketchAgent在设计、教育、工程等领域具有广泛应用前景,推动直观手绘与机器可读图表生成之间的鸿沟缩小。基准数据集已公开于https://huggingface.co/datasets/DiagramAgent/Sketch2Diagram-Benchmark。
原文摘要 · Abstract (English)
Hand-drawn sketches are a natural and efficient medium for capturing and conveying ideas. Despite significant advancements in controllable natural image generation, translating freehand sketches into structured, machine-readable diagrams remains a labor-intensive and predominantly manual task. The primary challenge stems from the inherent ambiguity of sketches, which lack the structural constraints and semantic precision required for automated diagram generation. To address this challenge, we introduce SketchAgent, a multi-agent system designed to automate the transformation of hand-drawn sketches into structured diagrams. SketchAgent integrates sketch recognition, symbolic reasoning, and iterative validation to produce semantically coherent and structurally accurate diagrams, significantly reducing the need for manual effort. To evaluate the effectiveness of our approach, we propose the Sketch2Diagram Benchmark, a comprehensive dataset and evaluation framework encompassing eight diverse diagram categories, such as flowcharts, directed graphs, and model architectures. The dataset comprises over 6,000 high-quality examples with token-level annotations, standardized preprocessing, and rigorous quality control. By streamlining the diagram generation process, SketchAgent holds great promise for applications in design, education, and engineering, while offering a significant step toward bridging the gap between intuitive sketching and machine-readable diagram generation. The benchmark is released at https://huggingface.co/datasets/DiagramAgent/Sketch2Diagram-Benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。