通过张量特征加速高维函数学习，显著降低存储开销。
Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network
- 在DNN首层引入可快速计算的秩-1特征与张量分解特征
- 采用随机张量分解，存储成本降低至少8个数量级
- 适用于5至40维函数的高效训练，适合高维建模任务
本文提出一种加速高维函数学习的方法，利用深度神经网络（DNN）进行优化。该方法在DNN第一层引入上下文特征，固定输入特征基底，通过标准梯度下降优化DNN参数；优化完成后，特征层可更新再继续训练。特征层包含两类函数：可在域上以矩阵无关方式快速评估的秩-1特征，以及需先通过张量网络（TN）分解策略处理的复杂特征。特别地，研究了将预训练DNN通过离散化与分解策略提炼为TN特征的效果。为高效分解由离散化DNN构建的高维函数，采用随机张量分解策略，使存储成本至少降低8个数量级。该方法可有效训练5至40维的模型。
原文摘要 · Abstract (English)
In this work we present a method to accelerate the optimization of learning high dimensional functions using deep neural network (DNN). This optimization procedure introduces contextual features into the first layer of a DNN. The parameters of DNN are optimized via standard gradient descent while keeping the input-feature basis fixed. After optimization of the DNN parameters, the feature layer is provided a chance to update and change before DNN optimization resumes. The feature layer has two types of functions: those that can be evaluated quickly in a matrix-free way on the domain (i.e. rank-1 features) and more complex features that must first be decomposed using tensor network (TN) decomposition strategies (tensor features). In particular, we study the effect of adding features which distill pretrained DNN into TNs using a discretize and decompose strategy. To efficiently decompose high-dimensional functions constructed from discretized DNN, we leverage a randomized tensor decomposition strategy. Using randomization, we are able to reduce the storage cost of decomposing high dimensional functions by at least 8 orders of magnitude. Using this approach, we are able to efficiently train models between 5 and 40 dimensions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。