用多模型验证和消幻机制，从体育新闻中自动提取准确且相关的赛前赛后洞察。
SUMMIR: A Hallucination-Aware Framework for Ranking Sports Insights from LLMs
- 构建双阶段验证流程，结合开源与私有大模型确保内容相关性。
- 在800场比赛的7900篇新闻上测试，发现不同模型事实一致性差异显著。
- 提出SUMMIR框架，按用户兴趣对洞察排序，提升个性化推荐效果。
随着在线体育新闻的快速普及，从比赛前后文章中提取有意义的见解对于提升用户参与度和理解力至关重要。本文针对自动提取比赛前后文章中的见解任务，构建了一个包含800场比赛、7900篇新闻的文章数据集，涵盖板球、足球、篮球和棒球四种主要运动。为保证上下文相关性，采用基于开源和专有大语言模型（LLMs）的两步验证流程。随后利用多个最先进的LLMs（GPT-4o、Qwen2.5-72B-Instruct、Llama-3.3-70B-Instruct、Mixtral-8x7B-Instruct-v0.1）生成全面的见解。通过基于FactScore的方法严格评估输出的事实准确性，并使用SummaC（摘要一致性）框架结合GPT-4o进行幻觉检测。最后，提出SUMMIR（Sentence Unified Multimetric Model for Importance Ranking）架构，根据用户特定兴趣对见解进行排序。实验表明该方法能生成高质量、相关性强的见解，同时揭示了不同模型在事实一致性和趣味性上的显著差异。本工作贡献了一个可靠的自动化体育新闻洞察生成框架。源代码已公开：https://github.com/nitish-iitp/SUMMIR。
原文摘要 · Abstract (English)
With the rapid proliferation of online sports journalism, extracting meaningful pre-game and post-game insights from articles is essential for enhancing user engagement and comprehension. In this paper, we address the task of automatically extracting such insights from articles published before and after matches. We curate a dataset of 7,900 news articles covering 800 matches across four major sports: Cricket, Soccer, Basketball, and Baseball. To ensure contextual relevance, we employ a two-step validation pipeline leveraging both open-source and proprietary large language models (LLMs). We then utilize multiple state-of-the-art LLMs (GPT-4o, Qwen2.5-72B-Instruct, Llama-3.3-70B-Instruct, and Mixtral-8x7B-Instruct-v0.1) to generate comprehensive insights. The factual accuracy of these outputs is rigorously assessed using a FactScore-based methodology, complemented by hallucination detection via the SummaC (Summary Consistency) framework with GPT-4o. Finally, we propose SUMMIR (Sentence Unified Multimetric Model for Importance Ranking), a novel architecture designed to rank insights based on user-specific interests. Our results demonstrate the effectiveness of this approach in generating high-quality, relevant insights, while also revealing significant differences in factual consistency and interestingness across LLMs. This work contributes a robust framework for automated, reliable insight generation from sports news content. The source code is availble here https://github.com/nitish-iitp/SUMMIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。