用神经网络直接学习递归效用下的动态规划解法,突破高维难题。
Deep Learning for Dynamic Programming with Recursive Utility

- 用神经网络联合学习价值函数、策略和确定性等价函数。
- 在高维问题中表现优异,误差控制在10^-4到10^-3之间。
- 适用于递归与期望效用模型,无需梯度或欧拉方程。
我们提出首个基于深度学习的求解高维离散时间递归效用动态规划问题的算法——确定性等价学习(CEL)。递归效用动态规划数值求解困难,因效用无显式表达,且贝尔曼方程包含难以计算的确定性等价项。CEL算法直接用神经网络学习该确定性等价值,并联合近似价值函数、策略函数和确定性等价函数。该方法无需网格、基于模拟,适用于高维状态与控制空间,不依赖欧拉方程、一阶条件或状态转移函数可微性。对于期望效用问题(作为递归效用的特例)同样有效。我们在折扣线性指数二次高斯控制、小噪声鲁棒控制、Epstein-Zin DSGE 及多资产战略配置问题上验证了该方法。相比闭式解与价值迭代基准,CEL在高维问题中仍保持高精度,小噪声鲁棒控制下准确度接近价值迭代,多数问题的样本外贝尔曼误差与欧拉/一阶残差在1.0e-4至1.0e-3区间。
原文摘要 · Abstract (English)
We propose the first deep learning algorithm, the Certainty Equivalent Learning (CEL) algorithm, for solving high-dimensional discrete-time dynamic programming problems with recursive utility. Dynamic programming with recursive utility is numerically challenging because the recursive utility does not have an explicit representation and the Bellman equation contains a certainty equivalent that is difficult to evaluate. The CEL algorithm learns this certainty-equivalent value directly with neural networks and jointly approximates value functions, policy functions, and certainty-equivalent functions. The CEL algorithm is mesh-free and simulation-based, allowing high-dimensional state and control spaces, and does not rely on Euler equations, first-order conditions, or differentiability of the state transition function. The CEL algorithm also works for dynamic programming problems with expected utility as expected utility is a special case of recursive utility. We apply the CEL to discounted linear exponential quadratic Gaussian control, small-noise robust control, Epstein-Zin DSGE, and multivariate strategic asset allocation problems. Compared with closed-form and VFI-based benchmarks, the CEL delivers accurate value and policy approximations, remains effective in high-dimensional problems, achieves accuracy comparable to VFI in the small-noise robust-control case, and produces out-of-sample Bellman errors and Euler or first-order residuals that are in the range from 1.0e-4 to 1.0e-3 for most problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。