让用户用草图、参考图和文字,一键生成并编辑风格统一的油画。
PaintFlow: A Unified Framework for Interactive Oil Paintings Editing and Generation
- 融合草图、参考图和文字,实现对油画风格的精准控制。
- 通过自监督风格迁移构建大规模配对数据集,解决油画数据稀缺问题。
- 采用AdaIN融合特征,确保生成图像风格一致,适合艺术创作与交互设计。
油画作为一种融合人类抽象思维与艺术表达的高阶媒介,其数字生成与编辑因笔触动态复杂和风格化特征显著而面临巨大挑战。现有方法受限于训练数据分布,多聚焦于真实照片的修改。本文提出一个统一的多模态框架,支持油画生成与编辑。用户可输入参考图像进行语义控制,手绘草图对齐空间结构,自然语言提示提供高层语义引导,同时保持输出风格统一。方法在训练阶段引入空间对齐与语义增强条件策略,将掩码与草图映射为空间约束,参考图与文本编码为特征约束,实现对象级语义对齐。为克服数据稀缺,提出基于笔触渲染(Stroke-Based Rendering, SBR)的自监督风格迁移流程,模拟油画修复的内补动态,将真实图像转化为保留笔触纹理的风格化油画,构建大规模配对训练数据集。推理阶段通过AdaIN操作整合多源特征,保障风格一致性。大量实验表明,该系统支持细粒度编辑,有效保留油画的艺术品质,在风格化油画生成与编辑中实现了前所未有的想象力实现水平。
原文摘要 · Abstract (English)
Oil painting, as a high-level medium that blends human abstract thinking with artistic expression, poses substantial challenges for digital generation and editing due to its intricate brushstroke dynamics and stylized characteristics. Existing generation and editing techniques are often constrained by the distribution of training data and primarily focus on modifying real photographs. In this work, we introduce a unified multimodal framework for oil painting generation and editing. The proposed system allows users to incorporate reference images for precise semantic control, hand-drawn sketches for spatial structure alignment, and natural language prompts for high-level semantic guidance, while consistently maintaining a unified painting style across all outputs. Our method achieves interactive oil painting creation through three crucial technical advancements. First, we enhance the training stage with spatial alignment and semantic enhancement conditioning strategy, which map masks and sketches into spatial constraints, and encode contextual embedding from reference images and text into feature constraints, enabling object-level semantic alignment. Second, to overcome data scarcity, we propose a self-supervised style transfer pipeline based on Stroke-Based Rendering (SBR), which simulates the inpainting dynamics of oil painting restoration, converting real images into stylized oil paintings with preserved brushstroke textures to construct a large-scale paired training dataset. Finally, during inference, we integrate features using the AdaIN operator to ensure stylistic consistency. Extensive experiments demonstrate that our interactive system enables fine-grained editing while preserving the artistic qualities of oil paintings, achieving an unprecedented level of imagination realization in stylized oil paintings generation and editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。