用隐式克罗内克结构加速高斯过程，实现高效学习曲线预测。
Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure
- 通过隐式克罗内克结构建模，融合高效核函数与缺失值处理。
- 计算复杂度降至O(n³ + m³)，内存需求为O(n² + m²)。
- 适合需要快速评估模型性能的AutoML场景，尤其关注早期停止。
AutoML中的关键任务是联合建模机器学习模型在不同超参数配置和训练进度下的学习曲线。尽管高斯过程（GPs）适用于此任务，但传统方法的时间复杂度为$\mathcal{O}(n^3m^3)$，空间复杂度为$\mathcal{O}(n^2 m^2)$，其中$n$为超参数配置数，$m$为每个配置的学习曲线观测数。通常，利用克罗内克结构实现高效推理与早期停止不兼容，因存在缺失学习曲线值。本文提出引入隐式克罗内克结构，将观测值的联合协方差矩阵视为潜在克罗内克积的投影。结合迭代线性求解器与结构化矩阵-向量乘法，新方法仅需$\mathcal{O}(n^3 + m^3)$时间与$\mathcal{O}(n^2 + m^2)$空间。实验表明，该GP模型在学习曲线预测任务上可达到Transformer的性能水平。
原文摘要 · Abstract (English)
A key task in AutoML is to model learning curves of machine learning models jointly as a function of model hyper-parameters and training progression. While Gaussian processes (GPs) are suitable for this task, naïve GPs require $\mathcal{O}(n^3m^3)$ time and $\mathcal{O}(n^2 m^2)$ space for $n$ hyper-parameter configurations and $\mathcal{O}(m)$ learning curve observations per hyper-parameter. Efficient inference via Kronecker structure is typically incompatible with early-stopping due to missing learning curve values. We impose $\textit{latent Kronecker structure}$ to leverage efficient product kernels while handling missing values. In particular, we interpret the joint covariance matrix of observed values as the projection of a latent Kronecker product. Combined with iterative linear solvers and structured matrix-vector multiplication, our method only requires $\mathcal{O}(n^3 + m^3)$ time and $\mathcal{O}(n^2 + m^2)$ space. We show that our GP model can match the performance of a Transformer on a learning curve prediction task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。