提出新算法,高效估算核岭回归中的重要列,实现单遍扫描、小内存计算。
Analysis of Nystrom method with sequential ridge leverage scores
- 逐次估算岭杠杆得分,用小规模数据摘要动态更新。
- 单遍扫描即可完成,空间需求仅依赖核矩阵的有效维度。
- 任意阶段都保证近似精度和模型统计风险,适合大规模核方法。
大规模核岭回归(KRR)受限于存储大型核矩阵 $K_t$ 的需求。为避免存储整个矩阵,Nystrom 方法通过采样核矩阵的子集列,基于重构矩阵高效求解近似 KRR。采样分布的选择影响统计与计算的权衡。近期研究显示,按岭杠杆得分(RLS)比例采样能提供强重构保证。尽管精确 RLS 难以计算,但可有效近似。本文研究序列设置下的 KRR 问题,提出 INK-ESTIMATE 算法,逐次估算 RLS。该算法维护一个小型 $K_t$ 摘要,在每一步用其生成中间的 RLS 估计。首先,摘要更新无需访问先前列,仅需单遍扫描;其次,算法所需空间固定且较小,仅依赖核矩阵的有效维度;最后,该摘要在任意步骤均提供对真实核矩阵与近似矩阵间距离,以及近似 KRR 解统计风险的强保证。
原文摘要 · Abstract (English)
Large-scale kernel ridge regression (KRR) is limited by the need to store a large kernel matrix K_t. To avoid storing the entire matrix K_t, Nystrom methods subsample a subset of columns of the kernel matrix, and efficiently find an approximate KRR solution on the reconstructed matrix. The chosen subsampling distribution in turn affects the statistical and computational tradeoffs. For KRR problems, recent works show that a sampling distribution proportional to the ridge leverage scores (RLSs) provides strong reconstruction guarantees for the approximation. While exact RLSs are as difficult to compute as a KRR solution, we may be able to approximate them well enough. In this paper, we study KRR problems in a sequential setting and introduce the INK-ESTIMATE algorithm, that incrementally computes the RLSs estimates. INK-ESTIMATE maintains a small sketch of K_t, that at each step is used to compute an intermediate estimate of the RLSs. First, our sketch update does not require access to previously seen columns, and therefore a single pass over the kernel matrix is sufficient. Second, the algorithm requires a fixed, small space budget to run dependent only on the effective dimension of the kernel matrix. Finally, our sketch provides strong approximation guarantees on the distance between the true kernel matrix and its approximation, and on the statistical risk of the approximate KRR solution at any time, because all our guarantees hold at any intermediate step.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。