arXiv:2505.24403cs.LG2025-05中稿 · ICLR

分析集合聚合函数的 Lipschitz 连续性,为集合型神经网络提供稳定性保障。

On the Lipschitz Continuity of Set Aggregation Functions and Neural Networks for Sets

  • 研究了求和、均值、最大值及注意力聚合在三种集合距离下的 Lipschitz 性。
  • 发现多数聚合函数仅对一种距离满足 Lipschitz 连续,注意力机制则全不满足。
  • 推导出集合神经网络的 Lipschitz 上界,适用于鲁棒性与分布外泛化分析。

神经网络的 Lipschitz 常数与其鲁棒性和泛化能力密切相关,因此估计模型的 Lipschitz 常数在多种场景下具有重要意义。以往研究主要集中在多层感知机和卷积神经网络上,本文聚焦于以集合或多重集向量形式建模的数据及相应的神经网络。这类模型通常使用置换不变的聚合函数(如求和、均值、最大值)或基于注意力的聚合函数,将输入多重集映射为单个向量。本文研究这些聚合函数在三种无序多重集距离度量下的 Lipschitz 连续性,并计算其 Lipschitz 常数。结果显示,在一般情况下,每种聚合函数仅对其中一种距离满足 Lipschitz 连续,而基于注意力的函数则对任意一种都不满足。在此基础上,本文推导出可处理向量多重集的神经网络的 Lipschitz 常数上界,并研究其对扰动的稳定性及在分布偏移下的泛化能力。通过在多个领域数据集上的实验验证了理论分析的有效性。

原文摘要 · Abstract (English)

The Lipschitz constant of a neural network is connected to several important properties of the network such as its robustness and generalization. It is thus useful in many settings to estimate the Lipschitz constant of a model. Prior work has focused mainly on estimating the Lipschitz constant of multi-layer perceptrons and convolutional neural networks. Here we focus on data modeled as sets or multi-sets of vectors and on neural networks that can handle such data. These models typically apply some permutation invariant aggregation function, such as the sum, mean or max operator, to the input multisets to produce a single vector for each input sample. In this paper, we investigate whether these aggregation functions, along with an attention-based aggregation function, are Lipschitz continuous with respect to three distance functions for unordered multisets, and we compute their Lipschitz constants. In the general case, we find that each aggregation function is Lipschitz continuous with respect to only one of the three distance functions, while the attention-based function is not Lipschitz continuous with respect to any of them. Then, we build on these results to derive upper bounds on the Lipschitz constant of neural networks that can process multisets of vectors, while we also study their stability to perturbations and generalization under distribution shifts. To empirically verify our theoretical analysis, we conduct a series of experiments on datasets from different domains.

Lipschitz集合神经网络稳定性分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。