arXiv:2605.22800cs.LGcs.AI2026-05

提出匹配原则,指导如何用训练惩罚减少部署时的模型漂移。

The Matching Principle: When Does a Training Penalty Cover Deployment Shift?

论文配图:The Matching Principle: When Does a Training Penalty Cover Deployment Shift?
图 1 · 摘自论文原文
  • 定义部署方向与训练惩罚的匹配关系,判断是否有效抑制漂移。
  • 均匀惩罚比无惩罚少30%以上部署漂移,线性模型中严格更优。
  • 已知重要方向时应针对性惩罚,否则保留残余漂移下限。

常规训练仅优化任务损失后停止,不关注内部表示能量:雅可比矩阵在未帮助标签的方向上可能保持较大值,导致微小保标签噪声即引发模型偏移——这是经典噪声注入理论仅在二阶下通过默认正则化修复的缺陷。本文提出匹配原则:将部署方向(Sigma_task)与训练惩罚(Sigma')对比,判断后者能否覆盖前者。默认的无思考策略是均匀分布/各向同性惩罚(Sigma' ∝ I),对应二阶高斯或岭回归,无需轴估计、不改架构,在简单线性岭模型中,其部署漂移严格低于仅任务训练,且无覆盖缺失。当轴已知时匹配更精确;若未知,则残留一个下限漂移。在七个领域中,已命名的二阶矩惩罚优于未正则化训练;受控实验验证了‘匹配 > 均匀 > 错误轴’的顺序。岭定理已证明,深层网络仍在指定扰动下为实验性质。设计准则:默认固定内部能量(均匀惩罚);已知轴时进行匹配;将控制表示敏感性的损失视为首要设计要素。

原文摘要 · Abstract (English)

Ordinary training optimises the task loss and then stops. It never pays for internal representation energy: Jacobians can stay large in directions that never helped the label, so even small label-preserving noise throws the model off---a design gap that classical noise-injection theory fixes at second order, but only when applied as default regularisation, which current practice does not do. We make that precise with a Matching Principle: name deployment directions (Sigma_task) and the training penalty Sigma', and ask whether the second covers the first. The no-thinking default is even-spread / isotropic penalty (Sigma' proportional to I)---classical Gaussian / Tikhonov at second order: no axis estimate, no architecture change, and---in a simple linear ridge model---strictly less deployment drift than task-only training, with no coverage miss by construction. When axes are known, matching is sharper; when they are missed, a residual floor remains. Across seven domains a named second-moment penalty beats unregularised training; a controlled illustration recovers match > even-spread > wrong-axis when axes are forced. The ridge theorems are proved; deep nets remain experiments under a specified perturbation. Design rule: fix internal energy by default (even-spread); match when axes are known; treat losses that control representation sensitivity as first-class design.

模型鲁棒性正则化训练策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。