揭示深度学习训练中记忆机制的影响与测量方法
Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gaps
- 从优化器状态、数据顺序等角度分析训练记忆来源
- 提出可移植的扰动工具和因果估计指标
- 适合关注模型可复现性与训练公平性的研究者
现代深度学习训练并非无记忆。更新依赖于优化器动量、平均策略、数据顺序策略（随机重排与有放回采样、分阶段增强与重放）、非凸路径以及辅助状态（教师模型EMA/SWA、对比队列、BatchNorm统计）。本文按来源、生命周期和可见性对记忆机制进行分类，引入种子配对函数空间因果估计量；设计可移植的扰动原语（如动量/Adam/EMA/BN的携带/重置、顺序窗口交换、队列/教师调整）；提出包含审计文档（顺序哈希、缓冲区/BN校验和、随机数生成器合约）的报告检查清单。结论是建立了一套可移植、因果、不确定性感知的测量协议，能量化不同模型、数据与训练范式下训练历史的影响程度。
原文摘要 · Abstract (English)
Modern deep-learning training is not memoryless. Updates depend on optimizer moments and averaging, data-order policies (random reshuffling vs with-replacement, staged augmentations and replay), the nonconvex path, and auxiliary state (teacher EMA/SWA, contrastive queues, BatchNorm statistics). This survey organizes mechanisms by source, lifetime, and visibility. It introduces seed-paired, function-space causal estimands; portable perturbation primitives (carry/reset of momentum/Adam/EMA/BN, order-window swaps, queue/teacher tweaks); and a reporting checklist with audit artifacts (order hashes, buffer/BN checksums, RNG contracts). The conclusion is a protocol for portable, causal, uncertainty-aware measurement that attributes how much training history matters across models, data, and regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。