在损失层引入双曲结构，让专家模型训练更稳定。
After the Euclidean Highway: Hyperbolic Expert AI as the Next Innovation
- 仅在损失层使用双曲几何，避免网络训练崩溃
- 六款专家小语言模型均成功训练，无一次数值溢出
- 适合需要稳定训练的专家系统开发者
专家领域具有树状结构，而欧氏变换器无法保持这种层级关系，导致深层信息衰减。以往研究关注如何弯曲网络，但未思考弯曲位置的影响。本研究发现：将双曲结构置于可训练适配器上会导致十七次训练崩溃（约220 GPU小时），而仅在损失层使用双曲结构则可稳定训练。提出HySAT（双曲结构感知训练）方法，仅在损失层采用双曲损失。在构建并部署的六款专家级小语言模型中（基于Llama 3.1和EXAONE 3.5；四种适配策略；1800万样本语料；317,000步优化器迭代无NaN），四臂对照实验验证了流形不变性的保留。三个命题与一个引理证明了为何仅损失层双曲化是稳定的，而适配器层双曲化不成立。四个模型已投入实际应用（一个上线服务），两个开源权重，所有训练过程轨迹及17起故障记录均已公开于Zenodo（CC-BY-4.0）。
原文摘要 · Abstract (English)
Expert domains are trees; the Euclidean transformer is not, diluting parent-child structure exponentially at depth. The hyperbolic turn left one question unasked: not how much of a network to curve, but where curvature may touch the gradient. Placement is a law, not a knob: the same geometry on a trainable adapter collapses training (seventeen training collapses, ~220 GPU-hours), yet at the loss layer alone it trains without one -- this is HySAT (Hyperbolic Structure-Aware Training), hyperbolic losses at the loss layer only. Across six expert SLMs we constructed and deployed (Llama 3.1 and EXAONE 3.5; four adapter strategies; 18.0M-sample corpus; zero NaN over ~317K optimizer steps), a matched four-arm ablation isolates the preserved manifold invariant, and three propositions and a lemma prove why loss-only placement is stable where adapter-on-manifold is not. Four models are operationally deployed (one live, consumer-facing), two open-weight, with per-step traces and a seventeen-incident failure ledger on Zenodo (CC-BY-4.0).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。