arXiv:2601.18064cs.LGcs.AI2026-01被引 3

受大脑启发的稀疏几何网络,用动态稀疏连接提升效率。

Resonant Sparse Geometry Networks

  • 在学习的双曲空间中构建稀疏连接,连接强度随距离衰减。
  • 长程依赖任务准确率达96.5%,参数量仅为Transformer的1/15。
  • 适合追求高效、生物可解释性模型的研究者使用。

我们提出共振稀疏几何网络(RSGN),一种受大脑启发的自组织稀疏分层输入依赖连接架构。与计算复杂度为O(n²)的密集注意力机制不同,RSGN将计算节点嵌入学习的双曲空间,连接强度随测地距离衰减,实现随输入动态调整的稀疏性。该架构运行在两个时间尺度:快速可微激活传播通过梯度下降优化,慢速类赫布学习结构适应通过局部相关性规则实现。严格数学分析表明,RSGN实现O(n*k)复杂度,其中k << n为平均活跃邻域大小。在层次分类和长程依赖任务上的实验表明,RSGN在长程依赖任务上达到96.5%准确率,参数量比标准Transformer少约15倍。在20类挑战性层次分类任务中,仅用41,672参数即达23.8%准确率(随机基线5%),远低于需403,348参数才能达到30.1%准确率的Transformer基线。消融实验确认各组件贡献,类赫布学习持续带来性能提升。结果表明,稀疏、几何组织的脑启发计算是更高效且生物合理神经架构的有前景方向。

原文摘要 · Abstract (English)

We introduce Resonant Sparse Geometry Networks (RSGN), a brain-inspired architecture with self-organizing sparse hierarchical input-dependent connectivity. Unlike Transformer architectures that employ dense attention mechanisms with O(n^2) computational complexity, RSGN embeds computational nodes in learned hyperbolic space where connection strength decays with geodesic distance, achieving dynamic sparsity that adapts to each input. The architecture operates on two distinct timescales: fast differentiable activation propagation optimized through gradient descent, and slow Hebbian-inspired structural learning for connectivity adaptation through local correlation rules. We provide rigorous mathematical analysis demonstrating that RSGN achieves O(n*k) computational complexity, where k << n represents the average active neighborhood size. Experimental evaluation on hierarchical classification and long-range dependency tasks demonstrates that RSGN achieves 96.5% accuracy on long-range dependency tasks while using approximately 15x fewer parameters than standard Transformers. On challenging hierarchical classification with 20 classes, RSGN achieves 23.8% accuracy (compared to 5% random baseline) with only 41,672 parameters, nearly 10x fewer than the Transformer baselines which require 403,348 parameters to achieve 30.1% accuracy. Our ablation studies confirm the contribution of each architectural component, with Hebbian learning providing consistent improvements. These results suggest that brain-inspired principles of sparse, geometrically-organized computation offer a promising direction toward more efficient and biologically plausible neural architectures.

稀疏网络双曲空间神经架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。