arXiv:2601.16139cs.LG2026-01中稿 · The 29th Internati…被引 1

揭示核学习中数据内在维度的两种度量,发现有效维度常远小于传统维度。

On the Intrinsic Dimensions of Data in Kernel Learning

  • 引入有效维度 $d_K$ 和 Minkowski 维度 $d_ρ$ 两种数据内在维数度量
  • 证明在大样本下误差界为 $O(n^{- rac{2+d_K}{2+2d_K} + ε})$
  • 提出采样算法可高效估计维度,适合研究复杂分布的模型泛化

流形假设认为,当输入分布支撑集的内在维度较小时,机器学习方法的泛化性能显著提升。本文在核岭回归(KRR)背景下研究两种内在维度的替代定义:$d_ρ$ 是基于核函数 $K$ 在域 $Ω$ 上诱导的度量的上 Minkowski 维度;$d_K$ 是由 $K$ 在 $Ω$ 上的 Kolmogorov $n$-宽度衰减速率导出的有效维度。针对 $Ω$ 上的概率测度 $μ$,分析了 $n$-宽度与积分算子 $ϕ→∫_ΩK(·,x)ϕ(x)dμ(x)$ 的特征值之间的关系。结果表明,对于固定域 $Ω$,Kolmogorov $n$-宽度刻画了所有支撑于 $Ω$ 的概率测度下最坏情况的特征值衰减。这些特征值是理解约束 KRR 泛化行为的核心,由此导出了当训练集大小 $n$ 较大时,误差界为 $O(n^{-\frac{2+d_K}{2+2d_K} + ε})$(任意 $ε > 0$)。此外,本文提出一种仅需从 $μ$ 中抽取有限样本即可估计 $n$-宽度上界的算法。对于接近均匀分布的情况,以高概率计算出 $ε$-准确的 $n$-宽度上界,所需样本数不超过 $O(ε^{-d_ρ}\log\frac{1}{ε})$,且 $n$ 越小所需样本越少。最后,对多种分形集计算了有效维度 $d_K$,并进行了数值实验。结果表明,对于拉普拉斯核等核函数,$d_K$ 可能显著小于 $d_ρ$,尽管在规则域上二者相等。

原文摘要 · Abstract (English)

The manifold hypothesis suggests that the generalization performance of machine learning methods improves significantly when the intrinsic dimension of the input distribution's support is low. In the context of KRR, we investigate two alternative notions of intrinsic dimension. The first, denoted $d_ρ$, is the upper Minkowski dimension defined with respect to the canonical metric induced by a kernel function $K$ on a domain $Ω$. The second, denoted $d_K$, is the effective dimension, derived from the decay rate of Kolmogorov $n$-widths associated with $K$ on $Ω$. Given a probability measure $μ$ on $Ω$, we analyze the relationship between these $n$-widths and eigenvalues of the integral operator $ϕ\to \int_ΩK(\cdot,x)ϕ(x)dμ(x)$. We show that, for a fixed domain $Ω$, the Kolmogorov $n$-widths characterize the worst-case eigenvalue decay across all probability measures $μ$ supported on $Ω$. These eigenvalues are central to understanding the generalization behavior of constrained KRR, enabling us to derive an excess error bound of order $O(n^{-\frac{2+d_K}{2+2d_K} + ε})$ for any $ε> 0$, when the training set size $n$ is large. We also propose an algorithm that estimates upper bounds on the $n$-widths using only a finite sample from $μ$. For distributions close to uniform, we prove that $ε$-accurate upper bounds on all $n$-widths can be computed with high probability using at most $O\left(ε^{-d_ρ}\log\frac{1}ε\right)$ samples, with fewer required for small $n$. Finally, we compute the effective dimension $d_K$ for various fractal sets and present additional numerical experiments. Our results show that, for kernels such as the Laplace kernel, the effective dimension $d_K$ can be significantly smaller than the Minkowski dimension $d_ρ$, even though $d_K = d_ρ$ provably holds on regular domains.

核方法内在维度泛化分析分形数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。