用标签代替链接,精准判断电报短消息真假
The Tag is the Signal: URL-Agnostic Credibility Scoring for Messages on Telegram
- 通过文本标签(主题/主张/行动号召/证据)评分,不依赖网址
- 在8.7万条消息上达到0.871的AUC与0.787的宏平均F1
- 标签法更少特征却更强泛化,适合罕见域名场景
Telegram已成为传播虚假信息的重要平台。现有方法多依赖域名信誉或词汇特征,但在电报上因消息短、无链接而失效。为此,我们提出TAG2CRED管道,直接根据文本标签评估可信度。设计了涵盖主题、主张类型、行动号召和证据的简洁标签体系,使用微调大模型打标签,并通过带L2正则的逻辑回归映射为[0,1]风险分。在87,936条关联Media Bias/Fact Check(MBFC)的数据上,采用网址掩码与域名独立划分验证,结果表明:模型ROC-AUC达0.871,宏平均F1为0.787,布里尔得分0.167,优于基线TF-IDF(宏平均F1 0.737,布里尔得分0.248);特征数更少,对低频域名泛化能力更强。堆叠集成模型(TF-IDF + TAG2CRED + SBERT)进一步提升性能,ROC-AUC达0.901,宏平均F1 0.813(布里尔得分0.114),表明风格标签与词汇特征可互补捕捉信息风险。
原文摘要 · Abstract (English)
Telegram has become one of the leading platforms for disseminating misinformational messages. However, many existing pipelines still classify each message's credibility based on the reputation of its associated domain names or its lexical features. Such methods work well on traditional long-form news articles published by well-known sources, but high-risk posts on Telegram are short and URL-sparse, leading to failures for link-based and standard TF-IDF models. To this end, we propose the TAG2CRED pipeline, a method designed for such short, convoluted messages. Our model will directly score each post based on the tags assigned to the text. We designed a concise label system that covers the dimensions of theme, claim type, call to action, and evidence. The fine-tuned large language model (LLM) assigns tags to messages and then maps these tags to calibrated risk scores in the [0,1] interval through L2-regularized logistic regression. We evaluated 87,936 Telegram messages associated with Media Bias/Fact Check (MBFC), using URL masking and domain disjoint splits. The results showed that the ROC-AUC of the TAG2CRED model reached 0.871, the macro-F1 value was 0.787, and the Brier score was 0.167, outperforming the baseline TF-IDF (macro-F1 value 0.737, Brier score 0.248); at the same time, the number of features used in this model is much smaller, and the generalization ability on infrequent domains is stronger. The performance of the stacked ensemble model (TF-IDF + TAG2CRED + SBERT) was further improved over the baseline SBERT. ROC-AUC reached 0.901, and the macro-F1 value was 0.813 (Brier score 0.114). This indicates that style labels and lexical features may capture different but complementary dimensions of information risk.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。