arXiv:2409.18694cs.CVcs.AI2024-09NeurIPS被引 1

无需标注数据,让AI像人脑一样自动关联视觉特征生成内容

Learning from Pattern Completion: Self-supervised Controllable Generation

  • 通过模块化自编码器实现功能分化,提升生成可控性
  • 自监督训练下,对涂鸦、素描等低质量输入仍具强泛化能力
  • 适合研究神经机制启发的生成模型或追求少标注场景的应用

人类大脑具有自发关联同一或相似视觉场景不同属性的能力,如将素描与现实物体联系起来,通常无需标注信息。相比之下,现有可控生成方法如ControlNet严重依赖深度图、语义分割图等标注数据,限制了可扩展性。受大脑皮层模块化与海马体模式补全机制启发,本文提出自监督可控生成(SCG)框架。首先在模块化自编码器中引入等变约束,促进模块间独立性与模块内相关性,实现功能分化;随后基于这些专用模块,采用自监督模式补全方式进行可控生成训练。实验表明,所提模块化自编码器能有效实现颜色、亮度、边缘检测等功能分化,并表现出方向选择性、色拮抗及中心-周围感受野等类脑特性。自监督训练使SCG自发具备关联生成能力,在绘画、素描和古代涂鸦等任务上表现优异。相比代表性方法ControlNet,SCG在高噪声场景下更具鲁棒性,且因自监督特性具备更强可扩展潜力。代码已开源于Github与Gitee。

原文摘要 · Abstract (English)

The human brain exhibits a strong ability to spontaneously associate different visual attributes of the same or similar visual scene, such as associating sketches and graffiti with real-world visual objects, usually without supervising information. In contrast, in the field of artificial intelligence, controllable generation methods like ControlNet heavily rely on annotated training datasets such as depth maps, semantic segmentation maps, and poses, which limits the method's scalability. Inspired by the neural mechanisms that may contribute to the brain's associative power, specifically the cortical modularization and hippocampal pattern completion, here we propose a self-supervised controllable generation (SCG) framework. Firstly, we introduce an equivariant constraint to promote inter-module independence and intra-module correlation in a modular autoencoder network, thereby achieving functional specialization. Subsequently, based on these specialized modules, we employ a self-supervised pattern completion approach for controllable generation training. Experimental results demonstrate that the proposed modular autoencoder effectively achieves functional specialization, including the modular processing of color, brightness, and edge detection, and exhibits brain-like features including orientation selectivity, color antagonism, and center-surround receptive fields. Through self-supervised training, associative generation capabilities spontaneously emerge in SCG, demonstrating excellent generalization ability to various tasks such as associative generation on painting, sketches, and ancient graffiti. Compared to the previous representative method ControlNet, our proposed approach not only demonstrates superior robustness in more challenging high-noise scenarios but also possesses more promising scalability potential due to its self-supervised manner.Codes are released on Github and Gitee.

自监督学习可控生成类脑模型生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。