用大模型+注意力机制提升文本分类,尤其改善长距离依赖和类别不平衡问题。
Advancing Text Classification with Large Language Models and Neural Attention Mechanisms
- 基于大模型生成深度语义嵌入,结合注意力机制聚焦关键特征。
- 在多个数据集上超越传统模型,召回率和AUC提升显著。
- 对超参数和类别不平衡有强适应性,适合复杂真实场景。
本研究提出一种基于大语言模型的文本分类算法,旨在解决传统方法在捕捉长距离依赖、理解上下文语义和处理类别不平衡方面的局限。框架包括文本编码、上下文表征建模、基于注意力的增强、特征聚合与分类预测。在表征阶段,通过大规模预训练语言模型获取深层语义嵌入,并应用注意力机制增强关键特征的选取;在聚合阶段,融合全局与加权策略生成鲁棒的文本级向量;在分类阶段,采用全连接层与Softmax输出预测类别分布,以交叉熵损失优化参数。对比实验引入循环神经网络、图神经网络和Transformer等基线模型,在精确率、召回率、F1分数和AUC指标上评估。结果表明,所提方法在所有指标上均优于现有模型,尤其在召回率和AUC上表现突出。此外,对超参数与数据条件进行敏感性实验,涵盖隐藏维度对AUC的影响及类别不平衡比对召回率的影响。结果显示,合理配置模型显著提升性能,揭示了模型在不同条件下的适应性与稳定性。总体而言,该方法不仅实现有效性能提升,还通过系统分析验证其在复杂数据环境中的鲁棒性与适用性。
原文摘要 · Abstract (English)
This study proposes a text classification algorithm based on large language models, aiming to address the limitations of traditional methods in capturing long-range dependencies, understanding contextual semantics, and handling class imbalance. The framework includes text encoding, contextual representation modeling, attention-based enhancement, feature aggregation, and classification prediction. In the representation stage, deep semantic embeddings are obtained through large-scale pretrained language models, and attention mechanisms are applied to enhance the selective representation of key features. In the aggregation stage, global and weighted strategies are combined to generate robust text-level vectors. In the classification stage, a fully connected layer and Softmax output are used to predict class distributions, and cross-entropy loss is employed to optimize model parameters. Comparative experiments introduce multiple baseline models, including recurrent neural networks, graph neural networks, and Transformers, and evaluate them on Precision, Recall, F1-Score, and AUC. Results show that the proposed method outperforms existing models on all metrics, with especially strong improvements in Recall and AUC. In addition, sensitivity experiments are conducted on hyperparameters and data conditions, covering the impact of hidden dimensions on AUC and the impact of class imbalance ratios on Recall. The findings demonstrate that proper model configuration has a significant effect on performance and reveal the adaptability and stability of the model under different conditions. Overall, the proposed text classification method not only achieves effective performance improvement but also verifies its robustness and applicability in complex data environments through systematic analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。