arXiv:2605.03762cs.AI2026-05被引 2

构建可复现的LLM预测评估框架,解决真实时间边界下的预测能力测试难题。

OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking

论文配图:OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking
图 1 · 摘自论文原文
  • 通过知识截止与时间掩码重建历史事件为时序预测样本
  • 将残留信息泄露降至1%,优于仅工具过滤的水平
  • 支持模型间公平对比,适配微调与强化学习训练

大语言模型正从静态文本生成转向现实世界的决策支持系统,其中预测能力整合了信息获取、证据融合、情境判断与行动决策。该能力在金融、政策、产业及科研中需求广泛,但评估困难:实时基准虽能准确衡量预测能力,但事件结束后即失效;回顾性基准可复现,却难以区分模型真实预测与预训练中已知事实。单纯提示模型‘假装不知’无法替代真实的知识边界。本文提出OracleProto,一个可复现的评估框架,通过模型截止对齐采样、工具级时间掩码、内容级泄漏检测、离散答案归一化与分层评分,将未来事件重构为受控时间边界的预测样本。在基于FutureX-Past的数据集上,使用六种主流LLM进行验证,OracleProto有效区分预测质量、采样稳定性与成本效率,同时将残余信息泄露降低至1%水平,较仅用工具过滤降低一个数量级。该框架使LLM预测能力评估从一次性测试升级为可审计、可复用、可训练的数据集级能力,提供统一接口用于跨模型公平比较,并为下游SFT与强化学习提供受控信号源。代码与数据见https://github.com/MaYiding/OracleProto 和 https://huggingface.co/datasets/MaYiding/OracleProto。

原文摘要 · Abstract (English)

Large language models are moving from static text generators toward real-world decision-support systems, where forecasting is a composite capability that links information gathering, evidence integration, situational judgment, and action-oriented decision making. This capability is in broad demand across finance, policy, industry, and scientific research, yet its evaluation remains difficult: live benchmarks evaluate forecasts before answers exist, making them the cleanest way to measure forecasting ability, but they expire once events resolve; retrospective benchmarks are reproducible, but they cannot reliably distinguish genuine forecasting from facts a model may have already learned during pretraining. Prompting models to "pretend not to know" cannot replace a genuine knowledge boundary. We propose OracleProto, a reproducible framework for evaluating LLM native forecasting capability. OracleProto reconstructs resolved events into time-bounded forecasting samples by combining model-cutoff-aligned sample admission, tool-level temporal masking, content-level leakage detection, discrete answer normalization, and hierarchical scoring. Instantiated on a FutureX-Past-derived dataset with six contemporary LLMs, OracleProto distinguishes forecasting quality, sampling stability, and cost efficiency under controlled information boundaries, while reducing residual leakage to the $1\%$ level, an order of magnitude below tool-only temporal filtering. OracleProto turns LLM forecasting from one-off evaluation into an auditable, reusable, and trainable dataset-level capability, providing a unified interface for fair cross-model comparison and a controlled signal source for downstream SFT and RL. Code and data are available at https://github.com/MaYiding/OracleProto and https://huggingface.co/datasets/MaYiding/OracleProto.

大模型评估时间建模预测能力可复现性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。