用历史数据回放测试大模型预测能力，避免信息泄露干扰评估。
Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
- 设定时间点 t₀ 冻结数据源，只允许模型读取 t₀ 前的公开内容
- 在真实预测市场中验证模型，得分同时对比实际结果与市场定价
- 可重复用于新模型和新事件，避免评估过时，适合关注预测可信度的研究者
评估预测模型通常采用回溯测试，即重演已解决的问题并评判系统在结果公布前给出的概率。对大语言模型而言，存在两个信息泄露渠道：一是模型检索可能获取事件发生后的报告，使预测变成查表；二是新模型训练数据更接近事件发生时间，导致过去未发生的事件已在当前数据中出现。这两种情况都让测试衡量的是记忆而非前瞻性判断。为此，本文提出 Hindcast，通过设定一个固定历史时间点 t₀，将模型置于 t₀ 时刻的“历史状态”下进行评估。该方法基于冻结的 Reddit 公共数据快照，仅允许模型读取在 t₀ 前发布的帖子，并以实际结果及 t₀ 时刻预测市场的价格作为双重评分标准。由于每个市场独立设定 t₀ 且快照不变，评估可随模型更新持续运行而不失效。实验发现，当 Reddit 事先讨论过事件时，检索仍能提升性能；若仅有猜测而无信息，则检索反而降低表现。
原文摘要 · Abstract (English)
Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date $t_0$, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before $t_0$, and scores each forecast against both what happened and the market's own price at $t_0$, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。