用信息论设计神经元局部目标,实现自组织学习。
What should a neuron aim for? Designing local objective functions based on information theory
- 基于部分信息分解(PID)为神经元设定局部学习目标。
- 神经元可自主选择输入信息的独有、冗余或协同贡献方式。
- 适合研究可解释性与高效本地化学习的算法设计者。
现代深度神经网络中,单个神经元的学习动态往往不清晰,因网络通过全局优化训练。而生物系统依赖自组织的局部学习,在有限全局信息下仍具鲁棒性和高效性。本文展示如何通过抽象的生物启发式局部学习目标,实现人工神经元间的自组织。这些目标利用信息论的新扩展——部分信息分解(PID),将一组信息源对结果的信息贡献分解为唯一、冗余和协同三部分。该框架使神经元能局部调控来自前馈、反馈和侧向输入的信息整合方式,即选择何种输入应以唯一、冗余或协同方式贡献输出。这一选择由PID项的加权和表达,针对具体任务可基于直觉推导或数值优化获得,为理解任务相关的局部信息处理提供了窗口。在保持强性能的同时实现神经元级可解释性,本工作建立了局部学习策略的原理性信息论基础。
原文摘要 · Abstract (English)
In modern deep neural networks, the learning dynamics of the individual neurons is often obscure, as the networks are trained via global optimization. Conversely, biological systems build on self-organized, local learning, achieving robustness and efficiency with limited global information. We here show how self-organization between individual artificial neurons can be achieved by designing abstract bio-inspired local learning goals. These goals are parameterized using a recent extension of information theory, Partial Information Decomposition (PID), which decomposes the information that a set of information sources holds about an outcome into unique, redundant and synergistic contributions. Our framework enables neurons to locally shape the integration of information from various input classes, i.e. feedforward, feedback, and lateral, by selecting which of the three inputs should contribute uniquely, redundantly or synergistically to the output. This selection is expressed as a weighted sum of PID terms, which, for a given problem, can be directly derived from intuitive reasoning or via numerical optimization, offering a window into understanding task-relevant local information processing. Achieving neuron-level interpretability while enabling strong performance using local learning, our work advances a principled information-theoretic foundation for local learning strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。