用真实决策时间数据测试大模型选股能力,发现实时通胀信息是关键。
Leakage-Aware Benchmarking of LLM Forecasting: Real-Time Nowcasts as the Decision-Time Input for Macro Factor Ranking
- 限定模型仅使用决策时可得数据,避免信息泄露干扰评估。
- 模型月度排名相关性中位数达+0.154,但整体统计效力不足。
- 大模型优势体现在极端排名,对投资组合构建更有价值。
检索增强型7B开源大模型在2023年4月至2026年3月每月末进行股权风格因子排序,仅使用决策时可得信息:滞后的FRED宏观变量、近期宏观事件摘要及克利夫兰联储存档的当日未发布月度通胀现报。通过宏观类比检索模块选取历史状态,批评者大模型将其压缩为一条战术规则,执行者大模型将当前状态与近期规则映射为七个美国股权风格因子得分。全链路在三个不重叠的12个月子窗口中均获得正平均秩相关性,中位数为+0.154;但整体均值统计效力不足,自举95%置信区间包含零。在相同约束下,非大模型基线表明kNN宏观类比模型可达到相近中位数相关性,说明实时通胀信息与宏观相似性检索已解释大部分信号。大模型管道在均值相关性与多空配置合理性检验上表现更优,表明其边际收益集中在驱动多空组合形成的极端排名上。附录包含36条批评者规则的描述性审计与逐月案例分析。
原文摘要 · Abstract (English)
Forecasting benchmarks for retrieval-augmented LLMs routinely confound model capability with information leakage: features labeled with a target's timestamp are often not observable at the system's decision time. We study leakage-controlled equity factor ranking with a retrieval-augmented 7B open-source LLM forecaster. At each month-end from 2023-04 to 2026-03, the forecaster observes only decision-time information: lag-shifted FRED macro variables, recent macro-event summaries, and the Cleveland Fed's archived daily CPI nowcast for unreleased current-month inflation. A macro-analog retrieval module selects historical states, a critic LLM compresses them into one tactical rule, and an actor LLM maps the current state and recent rules into scores for seven U.S. equity style factors. The full pipeline obtains a median monthly Spearman rank IC of +0.154, with positive means across three non-overlapping contiguous 12-month subwindows; the mean IC remains statistically underpowered, with a bootstrap 95% confidence interval that includes zero. Non-LLM baselines under the same decision-time constraint demonstrate that a kNN macro-analog model recovers a comparable median IC, indicating that real-time inflation information and macro-similar retrieval explain much of the median signal. The LLM pipeline retains higher mean IC and a stronger long-short allocation sanity check, suggesting that any marginal benefit is concentrated in the extreme rankings that drive long-short portfolio formation. A descriptive audit of the 36 critic rules and per-month case studies appears in the appendix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。