提出加权采样方法，显著减少随机特征模型所需特征数。
On the Generalization Properties of Learning the Random Feature Models with Learnable Activation Functions
- 采用数据依赖的加权采样生成特征，提升学习效率。
- 在均方误差下，特征数从1/ε²降至(1/ε)^{1/t}，低秩时可固定为常数。
- 适合追求高效特征选择的机器学习研究者和工程师。
本文研究了近期提出的可学习激活函数的随机特征模型（RFLAF）的泛化性质。通过引入数据依赖的采样方案，首次给出了回归与分类任务中学习RFLAF所需特征数的最紧界。统一理论揭示了特征数s的复杂性：对于普通采样，边界为Ω(1/ε²)；而采用加权采样后，在均方误差损失下可优化至˜Ω((1/ε)^{1/t})（t≥1），当核矩阵具有有限秩时甚至可降至Ω(1)。在Lipschitz损失情形，边界由Ω(1/ε²)改进为˜Ω((1/ε²)^{1/t})。为学习加权RFLAF，还提出了近似核构造与杠杆加权采样的算法。实验表明，加权RFLAF以更少特征达到相当性能，验证了理论有效性与方法实用性。
原文摘要 · Abstract (English)
This paper studies the generalization properties of a recently proposed kernel method, the Random Feature models with Learnable Activation Functions (RFLAF). By applying a data-dependent sampling scheme for generating features, we provide by far the sharpest bounds on the required number of features for learning RFLAF in both the regression and classification tasks. We provide a unified theorem that describes the complexity of the feature number $s$, and discuss the results for the plain sampling scheme and the data-dependent leverage weighted scheme. Through weighted sampling, the bound on $s$ in the MSE loss case is improved from $Ω(1/ε^2)$ to $\tildeΩ((1/ε)^{1/t})$ in general $(t\geq 1)$, and even to $Ω(1)$ when the Gram matrix has a finite rank. For the Lipschitz loss case, the bound is improved from $Ω(1/ε^2)$ to $\tildeΩ((1/ε^2)^{1/t})$. To learn the weighted RFLAF, we also propose an algorithm to find an approximate kernel and then apply the leverage weighted sampling. Empirical results show that the weighted RFLAF achieves the same performances with a significantly fewer number of features compared to the plainly sampled RFLAF, validating our theories and the effectiveness of this method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。