通过表征几何分析,揭示神经网络学习中超越懒惰-丰富二分法的多样化特征演化规律。
Feature Learning beyond the Lazy-Rich Dichotomy: Insights from Representational Geometry
- 从表征流形几何变化出发,刻画特征学习过程中的动态演化。
- 发现任务相关流形在学习中逐渐解纠缠,体现不同学习阶段与策略。
- 适用于理解神经科学中的结构归纳偏置与模型泛化机制。
将任务相关信息融入神经表征是生物与人工智能系统的核心能力。现有理论将学习分为两种范式:在丰富模式中,神经网络主动学习任务相关特征;在懒惰模式中,网络表现如随机特征模型。然而这一简单二分法忽略了由学习算法、网络架构和数据特性差异所塑造的复杂特征学习谱系。为填补此空白,本文提出一种基于神经表征几何的分析框架,不关注单个特征,而是考察任务相关表征流形在整个学习过程中的演化。理论与实证均表明,随着网络学习,任务相关流形逐渐解纠缠,其几何变化揭示了超越懒惰-丰富二分法的学习阶段与策略。该框架为神经科学与机器学习中的特征学习提供了新洞见,有助于理解神经回路中的结构归纳偏置及分布外泛化机制。
原文摘要 · Abstract (English)
Integrating task-relevant information into neural representations is a fundamental ability of both biological and artificial intelligence systems. Recent theories have categorized learning into two regimes: the rich regime, where neural networks actively learn task-relevant features, and the lazy regime, where networks behave like random feature models. Yet this simple lazy-rich dichotomy overlooks a diverse underlying taxonomy of feature learning, shaped by differences in learning algorithms, network architectures, and data properties. To address this gap, we introduce an analysis framework to study feature learning via the geometry of neural representations. Rather than inspecting individual learned features, we characterize how task-relevant representational manifolds evolve throughout the learning process. We show, in both theoretical and empirical settings, that as networks learn features, task-relevant manifolds untangle, with changes in manifold geometry revealing distinct learning stages and strategies beyond the lazy-rich dichotomy. This framework provides novel insights into feature learning across neuroscience and machine learning, shedding light on structural inductive biases in neural circuits and the mechanisms underlying out-of-distribution generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。