用预训练模型做假新闻检测，发现注意力编码效果稳定可靠
Pooling Attention: Evaluating Pretrained Transformer Embeddings for Deception Classification
- 固定预训练模型提取特征，搭配轻量分类器测试
- BERT+逻辑回归在LIAR数据集上超越神经网络基线
- 简单池化方法对截断有鲁棒性，适合快速部署
本文将假新闻检测作为Transformer表示的下游任务，评估仅编码器和仅解码器型预训练模型（BERT、GPT-2、Transformer-XL）在冻结状态下作为嵌入器的表现。通过控制预处理方式对比池化与填充、神经与线性分类头，结果表明上下文自注意力编码具有稳定的迁移能力。在LIAR数据集上，BERT嵌入结合逻辑回归优于神经基线；序列长度与聚合方式分析显示，模型对截断具有鲁棒性，最大值或平均池化均具优势。该研究将基于注意力的标记编码器定位为可信度任务的稳健架构基础，剥离了分类器复杂度对Transformer贡献的影响。
原文摘要 · Abstract (English)
This paper investigates fake news detection as a downstream evaluation of Transformer representations, benchmarking encoder-only and decoder-only pre-trained models (BERT, GPT-2, Transformer-XL) as frozen embedders paired with lightweight classifiers. Through controlled preprocessing comparing pooling versus padding and neural versus linear heads, results demonstrate that contextual self-attention encodings consistently transfer effectively. BERT embeddings combined with logistic regression outperform neural baselines on LIAR dataset splits, while analyses of sequence length and aggregation reveal robustness to truncation and advantages from simple max or average pooling. This work positions attention-based token encoders as robust, architecture-centric foundations for veracity tasks, isolating Transformer contributions from classifier complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。