提出稳定替换预训练模型模块的新方法，提升效率与收敛速度。
Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers
- 用确定性渐进权重融合教师与学生输出，避免梯度波动
- 单种子实验中收敛更快，对齐效果优于随机门控和蒸馏基线
- 适合需要高效替换注意力机制的模型优化场景
在预训练模型中替换模块（如将二次自注意力替换为高效替代方案）面临严峻优化挑战：冷启动重初始化会破坏冻结主干的稳定性。我们通过受控实验聚焦这一核心稳定性问题。确定性连续替换（DCR）采用确定性、渐进式权重融合教师与学生输出。理论上，DCR消除了随机替换中固有的门控梯度方差。在单种子实验中，DCR在受控注意力替换任务上实现了比随机门控和蒸馏基线更快的收敛速度与更强的对齐表现，为异构算子替换奠定了基础。
原文摘要 · Abstract (English)
Replacing modules in pretrained models, especially swapping quadratic self-attention for efficient attention alternatives, poses a hard optimization problem: cold-start reinitialization destabilizes frozen backbones. We isolate this core stability challenge in a controlled study. Deterministic Continuous Replacement (DCR) blends teacher and student outputs with a deterministic, annealed weight. Theoretically, DCR eliminates gate-induced gradient variance inherent to stochastic replacement. In a single-seed study, DCR attains faster convergence and stronger alignment than stochastic gating and distillation baselines on controlled attention replacement, establishing a foundation for heterogeneous operator swaps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。