通过分析文本预测新闻事实更新，提升大模型问答的可靠性。
NewsEdits 2.0: Learning the Intentions Behind Updating News
- 构建新闻编辑意图分类体系，区分事实、风格、叙事类修改
- 在9200+句对上训练模型，实现高精度事实更新预测
- 可用于大模型拒绝回答易过时问题，接近理想效果
随着事件发展，新闻文章常需更新信息：若不加警惕，可能传播过时事实。本文假设语言特征能反映事实动态性，并提出仅凭文章文本（无需外部资源）即可预测哪些事实会更新。我们首先在大规模新闻修订语料中识别事实更新，引入NewsEdits 2.0分类体系，将事实更新与风格、叙事更新区分开。对超过9,200对句子修订进行标注，并训练出高性能集成模型以应用该分类。随后，在大量银标签样本上，证明可高精度预测旧版稿件中事实的更新时机。最后，为验证实际价值，构建语言模型问答回避任务（LLM-QA abstention）。利用我们的预测结果，使大模型在信息可能过时时主动回避回答，准确率接近理想基准（oracle-level）。
原文摘要 · Abstract (English)
As events progress, news articles often update with new information: if we are not cautious, we risk propagating outdated facts. In this work, we hypothesize that linguistic features indicate factual fluidity, and that we can predict which facts in a news article will update using solely the text of a news article (i.e. not external resources like search engines). We test this hypothesis, first, by isolating fact-updates in large news revisions corpora. News articles may update for many reasons (e.g. factual, stylistic, narrative). We introduce the NewsEdits 2.0 taxonomy, an edit-intentions schema that separates fact updates from stylistic and narrative updates in news writing. We annotate over 9,200 pairs of sentence revisions and train high-scoring ensemble models to apply this schema. Then, taking a large dataset of silver-labeled pairs, we show that we can predict when facts will update in older article drafts with high precision. Finally, to demonstrate the usefulness of these findings, we construct a language model question asking (LLM-QA) abstention task. We wish the LLM to abstain from answering questions when information is likely to become outdated. Using our predictions, we show, LLM absention reaches near oracle levels of accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。