通过简化模型揭示概念学习中的动态规律,解释生成模型如何逐步掌握组合能力。
Swing-by Dynamics in Concept Learning and Compositional Generalization
- 构建结构化身份映射任务,模拟概念学习过程
- 理论分析发现早期测试损失非单调下降的新机制
- 适用于研究生成模型概念推理与泛化能力的学者
先前研究表明,文本条件扩散模型能够识别并操控构成数据生成过程的基本概念,实现对全新分布外组合的泛化。这些研究不仅评估性能,还揭示了学习动态的丰富现象:模型按组合层级顺序泛化,且数据中概念中心结构影响模型掌握概念操作的速度。本文从理论角度重新审视这些结果,提出结构化身份映射(SIM)任务——在具有结构性聚类中心的高斯混合上训练模型学习恒等映射。我们数学分析了神经网络在该任务上的学习动态,发现尽管形式简单,但其动态能捕捉并解释先前工作中观察到的关键现象。此外,理论还揭示了训练初期测试损失非单调下降的新机制,并通过训练文本条件扩散模型验证了预测。本工作将SIM任务确立为现代生成模型概念学习动态的重要理论抽象。
原文摘要 · Abstract (English)
Prior work has shown that text-conditioned diffusion models can learn to identify and manipulate primitive concepts underlying a compositional data-generating process, enabling generalization to entirely novel, out-of-distribution compositions. Beyond performance evaluations, these studies develop a rich empirical phenomenology of learning dynamics, showing that models generalize sequentially, respecting the compositional hierarchy of the data-generating process. Moreover, concept-centric structures within the data significantly influence a model's speed of learning the ability to manipulate a concept. In this paper, we aim to better characterize these empirical results from a theoretical standpoint. Specifically, we propose an abstraction of prior work's compositional generalization problem by introducing a structured identity mapping (SIM) task, where a model is trained to learn the identity mapping on a Gaussian mixture with structurally organized centroids. We mathematically analyze the learning dynamics of neural networks trained on this SIM task and show that, despite its simplicity, SIM's learning dynamics capture and help explain key empirical observations on compositional generalization with diffusion models identified in prior work. Our theory also offers several new insights -- e.g., we find a novel mechanism for non-monotonic learning dynamics of test loss in early phases of training. We validate our new predictions by training a text-conditioned diffusion model, bridging our simplified framework and complex generative models. Overall, this work establishes the SIM task as a meaningful theoretical abstraction of concept learning dynamics in modern generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。