让Transformer模拟胎儿视觉发育，发现其自发形成类脑结构。
Transformers self-organize like newborn visual systems when trained in prenatal worlds
- 用视网膜波生成器模拟胎儿期视觉输入，自监督训练Transformer。
- 模型早期层对边缘敏感，后期层对形状敏感，感受野逐层增大。
- 揭示神经网络与大脑共享学习机制，适合研究发育神经科学的学者。
Transformer 是否像大脑一样学习？一个关键挑战在于，大脑在出生前就通过视网膜波等感官体验“预训练”，而传统Transformer则在非生物合理的大型数据集上训练。我们推测，若两者学习方式相似，则在相同胎内视觉输入下应发展出类似的结构。为此，我们使用视网膜波生成器模拟胎儿期视觉输入，并采用自监督时间学习训练Transformer。训练过程中，模型自发形成了新生视觉系统的关键特征：（1）早期层对边缘敏感，（2）后期层对形状敏感，（3）各层感受野逐渐扩大。这表明，新生视觉系统的组织结构可在变压器适应胎内视觉世界时自发产生。这种发育上的收敛暗示大脑与变压器遵循相同的通用学习原则。
原文摘要 · Abstract (English)
Do transformers learn like brains? A key challenge in addressing this question is that transformers and brains are trained on fundamentally different data. Brains are initially "trained" on prenatal sensory experiences (e.g., retinal waves), whereas transformers are typically trained on large datasets that are not biologically plausible. We reasoned that if transformers learn like brains, then they should develop the same structure as newborn brains when exposed to the same prenatal data. To test this prediction, we simulated prenatal visual input using a retinal wave generator. Then, using self-supervised temporal learning, we trained transformers to adapt to those retinal waves. During training, the transformers spontaneously developed the same structure as newborn visual systems: (1) early layers became sensitive to edges, (2) later layers became sensitive to shapes, and (3) the models developed larger receptive fields across layers. The organization of newborn visual systems emerges spontaneously when transformers adapt to a prenatal visual world. This developmental convergence suggests that brains and transformers learn in common ways and follow the same general fitting principles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。