让交易模型在推理时动态优化,用预测价格提升收益与风险比。
Plan Before You Trade: Inference-Time Optimization for RL Trading Agents
- 推理时引入价格预测,构建目标函数动态优化策略
- 在DJ30基准上提升总回报与夏普、索提诺等风险调整指标
- 适配任意预训练模型,对随机策略效果更显著
用于投资组合管理的强化学习代理通常作为静态策略训练和部署,无法在推理时利用价格预测。我们提出FPILOT(金融插件推理时学习最优交易),受模型预测控制(MPC)启发。关键洞察是未来价格主要不依赖单个代理的资产配置,因此合适的预测模型可直接生成多步价格轨迹,无需传统强化学习中的迭代动作条件滚动。每个决策步骤中,我们利用预测的价格轨迹构建基于持仓的想象回报目标,并在推理时优化策略后再执行一步交易。该框架兼容任何预训练代理,无需重训练即可根据预测调整策略。在TradeMaster DJ30基准上,针对五种策略学习算法评估,FPILOT持续提升总回报及夏普、索提诺、卡尔玛等风险调整指标,随机策略收益提升更明显。进一步使用校准质量水平的合成预测表明,性能增益随预测质量提升而增加,说明未来金融预测进步将带动本方法表现持续改善。
原文摘要 · Abstract (English)
Reinforcement learning agents for portfolio management are typically trained and deployed as static policies, with no mechanism for using price forecasts at inference time. We propose $\text{FPILOT}$ (**Fin**ancial **P**lugin **I**nference-time **L**earning for **O**ptimal **T**rading), a plugin inference-time optimization framework inspired by Model Predictive Control (MPC). Our key structural insight is that future prices mostly do not depend on one agent's portfolio allocation, so a suitable predictive model can produce a multi-step price trajectory without iterative action-conditioned rollouts as in typical reinforcement learning. At each decision step, we use the forecaster's predicted price trajectory to construct an allocation-based imagined return objective, and optimize the policy at inference-time before executing one step of the trade. Our framework is compatible with any pre-trained agent and adapts the policy to the forecaster's predictions without any retraining. Evaluated across five policy learning algorithms on the TradeMaster DJ30 benchmark, $\text{FPILOT}$ produces consistent improvements in total return and return-based risk-adjusted metrics (Sharpe, Sortino, Calmar), with stochastic policies benefiting more than deterministic ones. Further, using synthetic forecasts at calibrated quality levels, we show that gains consistently improve with forecaster quality, suggesting that our performance will improve based on advances in financial forecasting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。