用神经网络检测新闻语料中的舆论突变点，自动识别重大事件影响。
Neural Total Variation Distance Estimators for Changepoint Detection in News Data
- 通过分类器混淆程度估算内容分布的总变差距离
- 在真实新闻数据中准确识别9/11、疫情等重大事件节点
- 无需领域知识，可量化分析公共话语变化，适合媒体与政策研究
检测公众话语在重大事件后的转变对理解社会动态至关重要。现实世界数据具有高维、稀疏和噪声大的特点，使该领域的变化点检测极具挑战性。本文利用神经网络进行新闻数据的变化点检测，提出一种基于‘学习-混淆’机制的方法，该机制最初用于探测物理系统中的相变。我们训练分类器以区分不同时间段的文章，利用分类准确率估计底层内容分布之间的总变差距离，显著的距离变化即指示变化点。我们在合成数据集和《卫报》真实新闻数据上验证了该方法的有效性，成功识别出9/11事件、新冠疫情及总统选举等重大历史事件。该方法所需领域知识极少，能自主发现公众话语的重大转变，并提供内容变化的量化度量，适用于新闻报道、政策分析与危机监测。
原文摘要 · Abstract (English)
Detecting when public discourse shifts in response to major events is crucial for understanding societal dynamics. Real-world data is high-dimensional, sparse, and noisy, making changepoint detection in this domain a challenging endeavor. In this paper, we leverage neural networks for changepoint detection in news data, introducing a method based on the so-called learning-by-confusion scheme, which was originally developed for detecting phase transitions in physical systems. We train classifiers to distinguish between articles from different time periods. The resulting classification accuracy is used to estimate the total variation distance between underlying content distributions, where significant distances highlight changepoints. We demonstrate the effectiveness of this method on both synthetic datasets and real-world data from The Guardian newspaper, successfully identifying major historical events including 9/11, the COVID-19 pandemic, and presidential elections. Our approach requires minimal domain knowledge, can autonomously discover significant shifts in public discourse, and yields a quantitative measure of change in content, making it valuable for journalism, policy analysis, and crisis monitoring.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。