揭示训练中误差信号分配如何形成隐式偏见,突破传统终点几何解释。
The Anatomy of Implicit Bias: Information Allocation in Neural Network Training
- 提出训练时信息分配视角,追踪误差信号在参数路径、坐标通道和样本区域的分布模式。
- 构建梯度需求、更新注入、移动平均增益等五项可观测诊断指标,量化信号分配过程。
- 适用于研究优化机制、设计可控训练策略的研究者,尤其关注训练动态与模型偏差关系。
隐式偏见通常被解释为优化过程对特定最终解及其几何结构的偏好。这一观点有助于理解模型为何停在某处,但未能直接说明偏见如何在训练中形成。本文提出一种训练时信息分配的新视角:优化过程在参数路径、坐标通道和样本区域上形成误差信号的书写模式。为此,本文构建了一套可观测的分配诊断工具,包括梯度需求、实际更新注入、指数移动平均引起的坐标增益、通道级更新比率和样本级损失分布。为分离训练进展与内部分配,引入坍缩-持续性分析:在匹配训练损失下,若外部损失统计坍缩而内部分配比例仍保持分离,则表明该因素改变了训练信号的内部分配。总体而言,本文将隐式偏见的分析从终态几何扩展至训练时信号分配。核心主张是:隐式偏见不仅体现在最终解,更体现在训练过程中哪些参数路径、坐标通道和样本区域优先且更强地接收误差信号。基于此,本文将不同训练因素纳入统一的信息分配诊断框架,提供机制层面的解释,并为未来可分别控制训练进度与信号分配的优化方法奠定基础。
原文摘要 · Abstract (English)
Implicit bias is usually explained as the preference of an optimization process for certain final solutions and their geometry. This view helps explain where a model finally stops. It gives less direct explanation of how this bias is formed during training. This paper proposes a training-time information allocation view. Under this view, optimization forms a writing pattern for error signals across parameter paths, coordinate channels, and sample regions. This paper builds a set of observable allocation diagnostics. These diagnostics include gradient demand, actual update injection, coordinate gain induced by exponential moving averages, channel-level update ratios, and sample-wise loss distributions. To separate training progress from internal allocation, this paper introduces a collapse--persistence analysis. Under matched training loss, if external loss statistics collapse but internal allocation ratios remain separated, then the factor changes the internal allocation of the training signal. Overall, this paper extends the analysis of implicit bias from final-solution geometry to training-time signal allocation. The main claim is that implicit bias is not only reflected by the final solution. It is also reflected by which parameter paths, coordinate channels, and sample regions receive the error signal first and more strongly during training. Based on this view, this paper places different training factors into a unified information-allocation diagnostic framework. The framework gives a mechanism-level explanation of training-time implicit bias. It also provides a basis for future optimization methods that control training progress and signal allocation separately.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。