构建韩语政治话语新数据集，验证了提示设计对攻击性语言检测的有效性。
From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse
- 基于三种优化判断构建伪真实标签，提升评估可靠性。
- 单次提示策略表现接近高成本方法，效率更高。
- 适合资源受限场景下的真实应用，尤其关注语言演化问题。
尽管攻击性语言持续演变，现有基于大模型的研究仍主要依赖过时数据集，且很少评估在未见文本上的泛化能力。本研究构建了一个大规模当代韩语政治话语数据集，并在无真实标签的情况下采用三种精细化判断，每种对应一种代表性检测方法且经过最优条件设计。通过留一法分析标签一致性，识别出各判断的独特模式。以伪标签作为近似真实标准进行量化评估，发现经策略设计的单次提示即可达到与更复杂方法相当的效果，表明该方法在现实约束条件下具备可行性。
原文摘要 · Abstract (English)
Although offensive language continually evolves over time, even recent studies using LLMs have predominantly relied on outdated datasets and rarely evaluated the generalization ability on unseen texts. In this study, we constructed a large-scale dataset of contemporary political discourse and employed three refined judgments in the absence of ground truth. Each judgment reflects a representative offensive language detection method and is carefully designed for optimal conditions. We identified distinct patterns for each judgment and demonstrated tendencies of label agreement using a leave-one-out strategy. By establishing pseudo-labels as ground trust for quantitative performance assessment, we observed that a strategically designed single prompting achieves comparable performance to more resource-intensive methods. This suggests a feasible approach applicable in real-world settings with inherent constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。