提出新型神经网络，实现单批次持续学习且内存高效。
p-Spin Glass Network Efficient Single-Batch Continual Learning
- 采用p自旋玻璃结构，原生三值量化压缩参数8倍
- 仅需1/8训练序列即达Transformer性能，支持微批次为1的稳定训练
- 适用于文本与原始字节流，适合边缘AI和持续学习场景
现代序列模型依赖大量内存和大批次随机优化，限制了样本效率和持续学习。本文提出p-自旋玻璃网络，克服这些限制，结构性控制优化方差，具备四项显著能力：1. 内存高效：原生三值量化使内部参数压缩8倍，精确隐式梯度将激活内存严格限制在$/mathcal{O}(B imes T imes D)$；2. 样本高效：在仅使用1/8训练序列的情况下达到Transformer基线的渐近性能；3. 支持单批次稳定性，微批次大小为1时仍能实现平稳、单调收敛；4. 稳定性具有模态无关性，在离散子词与长时序原始字节流上均保持鲁棒的时间信用分配。本工作消除了深度学习对大批次的需求，为持续学习与边缘AI奠定基础。
原文摘要 · Abstract (English)
Modern sequence models heavily rely on massive memory footprints and large-batch stochastic optimization, barriers that restrict sample efficiency and continual learning. We introduce the $p$-Spin Glass Network, a novel architecture that overcomes these limitations, structurally manages optimization variance and yields four noticeable capabilities: 1. It enforces memory efficiency: native ternary quantization compresses internal parameters by $8\times$, while exact implicit gradients strictly bound activation memory to $\mathcal{O}(B \cdot T \cdot D)$. 2. it demonstrates sample efficiency, matching the asymptotic performance of a Transformer baseline while utilizing $8\times$ fewer training sequences. 3. Method enables single-batch stability and smooth, monotonic convergence at a stochastic micro-batch size of $1$. 4. Finally, this stability proves modality-agnostic, maintaining robust temporal credit assignment across both discrete subword and long horizon uncompressed raw byte streams. Ultimately, this work removes large batch requirement for stable deep learning, establishing a foundation for continuous learning and edge AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。