测试大模型补全被遮蔽句子的能力，发现其在非结构化文本中表现不佳。
What am I missing here?: Evaluating Large Language Models for Masked Sentence Prediction
- 用遮蔽句子预测任务评估模型对长距离上下文的理解能力
- 在维基百科等非结构化文本中，模型补全准确率低于40%
- 适合关注大模型全局语义理解短板的研究者阅读
基于Transformer的模型主要依赖下一词预测（NTP），仅关注单个词的局部流畅性，缺乏对跨句连贯性的显式激励。为探究大模型在长距离语义预测上的能力，我们评估了GPT-4o、Claude 3.5 Sonnet和Gemini 2.0 Flash三款商用大模型在三种领域（ROCStories叙事、Recipe1M流程、Wikipedia说明）的遮蔽句子预测（MSP）任务表现，衡量补全句子的忠实度与上下文契合度。结果显示，尽管这些模型在其他任务上表现优异，在非结构化领域（如维基百科）的句子补全准确率低于40%，暴露出当前模型在全局连贯性建模上的显著缺陷。
原文摘要 · Abstract (English)
Transformer-based models primarily rely on Next Token Prediction (NTP), which predicts the next token in a sequence based on the preceding context. However, NTP's focus on single-token prediction often limits a model's ability to plan ahead or maintain long-range coherence, raising questions about how well LLMs can predict longer contexts, such as full sentences within structured documents. While NTP encourages local fluency, it provides no explicit incentive to ensure global coherence across sentence boundaries-an essential skill for reconstructive or discursive tasks. To investigate this, we evaluate three commercial LLMs (GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash) on Masked Sentence Prediction (MSP) - the task of infilling a randomly removed sentence - from three domains: ROCStories (narrative), Recipe1M (procedural), and Wikipedia (expository). We assess both fidelity (similarity to the original sentence) and cohesiveness (fit within the surrounding context). Our key finding reveals that commercial LLMs, despite their superlative performance in other tasks, are poor at predicting masked sentences in low-structured domains, highlighting a gap in current model capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。