模型可自生长结构,实现持续学习中稳定扩展。
SoftModel: A Neural Model That Grows Its Own Topology -- Governed Structural Growth for Continual In-Service Learning
- 通过可审计的结构算子动态调整模型宽度、层级和注意力等结构。
- 在持续学习任务中保持长期能力,避免固定结构带来的性能瓶颈。
- 适合需要长期演进的生产环境系统,如实时数据流处理。
当前神经网络通常分为训练与部署两个阶段,且结构一旦确定便不再变化。本文提出完全可塑性假设:模型结构本身也应是可变的,不被冻结。由此设计出一种可自我生长的软模型(SoftModel),其结构变化由一个独立的现实门控机制统一决策,涵盖宽度、层次、组合、输入接口、循环结构及注意力头等操作,每一步都精确控制、预算约束并可审计。整个系统由生产级大模型端到端驱动,实现从零构建。实验表明,在持续学习场景下,该机制能有效维持长期学习能力;结构随需求增长,打破了固定拓扑对后期能力的隐性限制。此外,新容量的边际价值在采纳前不可观测,因此治理机制采用事后评估形式。该机制同样适用于评估信号,核心方法在标准持续学习基准上验证有效,能支持长序列任务学习。预注册实验计划公开报告了正负结果,完整地图构成主要贡献。
原文摘要 · Abstract (English)
Today, a neural system is almost always used in two phases -- trained, then deployed -- and in that regime it freezes twice: training ends, and the topology itself was never a degree of freedom. We take the opposite premise as an axiom -- total plasticity: no part of a model, including its structure, is ever frozen -- and derive the governance a lifelong learner then requires. The design's target regime is continual, in-service learning: a long-lived model on a non-stationary stream, whose stability comes from governance rather than immobility and whose capacity follows demand. The result is a growable soft model: an algebra of structural operators (width, hierarchy, composition, input interface, grown cycles, attention heads), each exact at application, budgeted, and audited, with adoption decided solely by a held-out reality gate that treats parametric and structural change uniformly. A complete from-scratch system realizes the whole account; its factory surface is operated end-to-end by a production LLM. Two conclusions follow from the axiom by construction: stability under lifelong change becomes an audit property of the lifecycle, and structure that follows demand removes the silent cap a fixed topology places on later capability where the capacity floor binds. A third is measured: in the worlds where this was measured, the marginal value of new capacity was unobservable before adoption, so workable growth governance took its ex-post form. The same governance extends to evaluative signals, and the core method is evaluated on standard continual-learning benchmarks, where governed growth preserves the ability to keep learning along long task sequences. A pre-registered experimental program adjudicates the mechanism and value claims on the tested problems and reports its failures at full prominence; the map -- positive and negative -- is the contribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。