用大模型生成假新闻标题，替代真实数据做情感分析。
LLM-Generated Negative News Headlines Dataset: Creation and Benchmarking Against Real Journalism
- 用定制提示词生成涵盖多领域负面情绪的合成新闻标题。
- 合成标题在语义、语气、长度上与真实标题高度相似，仅专有名词占比略有差异。
- 适合需要隐私安全数据集的研究者，尤其关注情感分析与生成内容评估。
本研究探讨大型语言模型（LLM）生成数据集在自然语言处理任务中的潜力，旨在解决真实数据获取困难和隐私问题。聚焦负面情绪文本这一情感分析的关键部分，我们探索使用LLM生成的合成新闻标题作为真实数据的替代方案。通过定制化提示词，构建了一个涵盖多种社会领域负面情绪的专用语料库。合成标题经专家评审，并在嵌入空间中分析其与真实负面新闻在内容、语气、长度和风格上的对齐程度。评估指标包括与真实标题的相关性、困惑度、连贯性和真实性。合成数据集与两组真实新闻标题进行对比测试，涵盖比较困惑度测试、可读性测试、词性分布分析、BERTScore及语义相似性比较。结果显示，生成标题与真实标题高度匹配，唯一显著差异出现在词性分布测试中的专有名词占比。
原文摘要 · Abstract (English)
This research examines the potential of datasets generated by Large Language Models (LLMs) to support Natural Language Processing (NLP) tasks, aiming to overcome challenges related to data acquisition and privacy concerns associated with real-world data. Focusing on negative valence text, a critical component of sentiment analysis, we explore the use of LLM-generated synthetic news headlines as an alternative to real-world data. A specialized corpus of negative news headlines was created using tailored prompts to capture diverse negative sentiments across various societal domains. The synthetic headlines were validated by expert review and further analyzed in embedding space to assess their alignment with real-world negative news in terms of content, tone, length, and style. Key metrics such as correlation with real headlines, perplexity, coherence, and realism were evaluated. The synthetic dataset was benchmarked against two sets of real news headlines using evaluations including the Comparative Perplexity Test, Comparative Readability Test, Comparative POS Profiling, BERTScore, and Comparative Semantic Similarity. Results show the generated headlines match real headlines with the only marked divergence being in the proper noun score of the POS profile test.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。