用对数宽度的ReLU网络可实现抗扰动记忆，且宽度下限已明确。
Logarithmic Width Suffices for Robust Memorization
- 采用对数宽度的前馈ReLU网络实现鲁棒记忆
- 在任意lp范数下，宽度需为log(N)级才能保证独立于N的抗扰能力
- 适用于需要高鲁棒性记忆的场景，如对抗训练
神经网络的记忆容量在多项研究中已被深入探讨。众所周知，仅需固定宽度即可记忆N个样本，与N无关。然而，现有构造通常极为精细。本文关注前馈ReLU网络在给定扰动半径下的鲁棒记忆能力。我们建立了lp范数下可能扰动半径的上下界，表明在一般lp范数下，达到独立于样本数N的鲁棒记忆，所需宽度必须为log(N)量级，且该量级亦充分。这意味着，对数宽度足以实现鲁棒记忆。
原文摘要 · Abstract (English)
The memorization capacity of neural networks with a given architecture has been thoroughly studied in many works. Specifically, it is well-known that memorizing $N$ samples can be done using a network of constant width, independent of $N$. However, the required constructions are often quite delicate. In this paper, we consider the natural question of how well feedforward ReLU neural networks can memorize robustly, namely while being able to withstand adversarial perturbations of a given radius. We establish both upper and lower bounds on the possible radius for general $l_p$ norms, implying (among other things) that width logarithmic in the number of input samples is necessary and sufficient to achieve robust memorization (with robustness radius independent of $N$).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。