arXiv:2608.27857cs.AI2026-08

提出稳定自回归脉冲语言模型迁移方法,解决训练漂移问题。

SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models

论文配图:SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models
图 1 · 摘自论文原文
  • 在自生成前缀上持续监督,保持推理稳定性
  • 三种规模模型准确率提升0.8至2.9点
  • 适合需要低功耗语言建模的部署场景

脉冲神经网络(SNN)通过稀疏编码和事件驱动计算实现高效能语言建模,但从头训练高性能的SNN仍具挑战。现有方法通过知识蒸馏(KD)将预训练的人工神经网络(ANN)教师迁移到SNN学生,但传统方法仅在固定语料前缀上蒸馏,而自回归推理依赖自生成前缀,导致前缀源不匹配。这表现为输出策略与教师不一致,以及自生成与匹配前缀间脉冲动力学漂移。本研究提出一种稳定的在线策略蒸馏框架SpikeOPD,使SNN在自生成前缀上持续学习,同时引入全KL教师校正以缓解输出策略偏差,通过匹配前缀策略锚定约束策略偏离,并采用分层脉冲正则化控制放电率变化。在三个模型规模下,相比对应的传统KD-SNN,SpikeOPD在0.125B、0.35B和1.3B参数量时平均准确率分别提升0.8、1.7和2.9点,且保持稀疏计算特性。

原文摘要 · Abstract (English)

Spiking neural networks (SNNs) offer a path to energy-efficient language modeling through sparse encoding and event-driven computation, but training capable spiking language models from scratch remains difficult. A practical alternative is ANN-to-SNN migration through knowledge distillation (KD), where a pretrained artificial neural network (ANN) teacher supervises an SNN student. Existing migration approaches distill on fixed corpus prefixes, whereas autoregressive inference conditions on self-generated prefixes, creating prefix-source mismatch. It manifests as output-policy mismatch with the ANN teacher and internal spiking-dynamics drift between self-generated and matched corpus prefixes. On-policy distillation (OPD) offers a natural way to mitigate both manifestations by continuing teacher supervision on self-generated prefixes. We evaluate a teacher-only full-KL variant, Vanilla OPD, via a controlled stress test and observe it may suffer from delayed rollout-feedback collapse. This result shows that on-policy coverage alone does not ensure stable adaptation. Motivated by these findings, we propose SpikeOPD, a stable on-policy distillation framework for autoregressive SNNs that learns from self-generated prefixes while maintaining rollout stability. It applies full-KL teacher correction to reduce output-policy mismatch, while matched-prefix policy anchoring constrains policy departure from the frozen reference SNN on the same prefixes. Layerwise spike regularization further limits firing-rate deviations during on-policy adaptation. Across three model scales, SpikeOPD improves average accuracy over the corresponding KD SNNs by 0.8, 1.7, and 2.9 points at 0.125B, 0.35B, and 1.3B, respectively, while preserving their sparse-compute profiles.

脉冲神经网络知识蒸馏语言建模低功耗

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。