用数学公式续写任务测试模型预测能力,发现大模型能超越表面文本直接预测。
Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities

- 基于公式后缀预测构建无标签自监督基准,评估模型是否真正理解上下文
- 1363个公式续写中,大模型预测显著提升似然得分,且可区分模型层级与推理强度
- 首次揭示模型预测对评分器的干扰机制,适合用于检测训练中的捷径漏洞
我们提出一个自动生成的基准,用于预测技术论文中的隐藏文本。给定可见上下文 $X$ 与隐藏续写 $Y$,模型生成辅助预测 $Z$,再由独立评分器在有无 $Z$ 的条件下计算 $Y$ 的下一个词概率。该设计可无标签检验 $Z$ 是否传递有效信息,对比以近期上下文替代 $Z$ 的控制组。主要测试任务为方程后缀预测:模型看到上下文及公式前半部分,需预测后半部分。该任务融合表面文本建模与推理敏感推断;因存在多个等价续写,需统计分析而非逐项判断。在138篇近期物理与数学论文中的1363个方程续写上,GPT-5.5、Opus 4.7与GPT-5.4 nano的预测均优于上下文控制组,在Qwen3-8B与Kimi K2.6评分器下表现一致。通过微调评分器仅使用上下文提示并应用于保留论文,形成更强控制;此时GPT-5.5仍胜出,而GPT-5.4 nano未胜。更长的段落/TeX续写也呈现正向但噪声较大的提升,集中在目标开头。结果支持跨模型似然评分作为静态基准,并可用于在强化学习或模型选择前探测捷径漏洞。
原文摘要 · Abstract (English)
We introduce an automatically generated benchmark for predicting hidden text in technical papers. A paper supplies visible context $X$ and a hidden continuation $Y$; the evaluated model writes an auxiliary forecast string $Z$, and a separate scorer assigns next-token probability to $Y$ both with and without conditioning on $Z$. This gives a label-free test of whether $Z$ transmits information about the continuation, compared against controls where $Z$ is recent context rather than a forecast. Our main testbed is equation-suffix prediction: the predictor sees context and the first part of a displayed equation, then forecasts the rest. The task mixes surface-level arXiv/TeX text modeling with reasoning-sensitive inference; the suffix is one of many roughly equivalent continuations, so the benchmark is read statistically rather than item-by-item. On 1363 equation continuations from 138 recent physics and mathematics papers, forecasts from GPT-5.5, Opus 4.7, and GPT-5.4 nano all improve clipped likelihood over the context control under both Qwen3-8B and Kimi K2.6 scorers, distinguishing model families and reasoning-effort settings without human labels. To emulate shortcuts where $Z$ further primes the scorer rather than making a useful forecast, we also fine-tune the scorer on context-only prompts and apply it to held-out papers as a stronger control. GPT-5.5 forecasts still beat this fine-tuned control; GPT-5.4 nano forecasts do not. Longer prose/TeX continuations show positive but noisier lift over controls, concentrated near the beginning of the target. These results support cross-model likelihood scoring as a static benchmark and as a setup for probing shortcut vulnerabilities before reinforcement learning or model-selection optimization is applied.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。