arXiv:2606.08379cs.AIcs.CE2026-06

新强化学习模型优化大额股票卖出,降低交易滑点。

TT-DAC-PS: Twin-Target Deterministic Actor-Critic with Policy Smoothing for Optimal Trade Execution

  • 用双目标网络+策略平滑抑制价值过估计
  • 在10只美股上平均滑点降低,方差可控
  • 适合量化交易、算法执行等场景

本研究针对大额股票卖出程序的最优执行问题,提出TT-DAC-PS(双目标确定性演员-评论家带策略平滑)模型。该模型结合双指数移动平均评论家目标与悲观最小值备份、TD3风格的目标策略平滑噪声、延迟演员更新及保守Q正则化,有效抑制价值过估计。探索阶段采用奥恩斯坦-乌伦贝克(OU)噪声,结合周期性衰减、基于近期收益离散度的自适应调整及可学习的温度参数映射至噪声尺度。环境融合阿尔姆格伦-克里斯(AC)交易影响、限价订单簿(LOB)价格与成交量,状态特征归一化,每步交易量上限限制,以及基于效用的奖励函数。在10只美国股票的订单簿数据上测试,性能优于PPO、SAC、A2C等强化学习基线,以及TWAP、VWAP和AC等经典执行算法,平均实现更低的实施滑点百分比,且方差表现良好。

原文摘要 · Abstract (English)

This study addresses the optimal execution of large stock sell programs by introducing TT-DAC-PS (Twin-Target Deterministic Actor-Critic with Policy Smoothing), a deterministic actor-critic architecture that combines twin exponential-moving-average critic targets with pessimistic min backup, TD3-style target policy smoothing noise, delayed actor updates, and conservative Q regularisation to curb overestimation. Exploration uses Ornstein-Uhlenbeck (OU) noise with a hybrid schedule: deterministic episode-wise decay, variance-guided adjustment based on recent reward dispersion, and a Soft Actor-Critic (SAC)-style temperature that is learned and mapped to the noise scale. The environment integrates Almgren-Chriss (AC) trade impact with Limit Order Book (LOB) prices and volumes, normalised state features, per-step volume participation caps, and a utility-based reward. The trade execution algorithm is applied to LOB data for ten U.S. stocks. Performance is assessed against reinforcement-learning baseline algorithms, including Proximal Policy Optimisation (PPO), Soft Actor-Critic (SAC), and Advantage Actor-Critic (A2C), as well as alternative trade execution algorithms, including Time-Weighted Average Price (TWAP), Volume-Weighted Average Price (VWAP), and AC. The proposed model consistently reduces mean implementation shortfall percentage with competitive variance, outperforming classical baselines and standard reinforcement-learning benchmark models.

强化学习交易执行量化交易

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。