用图结构引导语言模型微调，提升语义表征质量
G-Loss: Graph-Guided Fine-Tuning of Language Models
- 构建文档相似性图，利用全局语义关系指导训练
- 在5个基准数据集上收敛更快，准确率更高
- 适合需要高质量文本表征的分类任务
传统损失函数（如交叉熵、对比损失）在微调预训练语言模型（如BERT）时仅关注局部邻域，忽略全局语义结构。本文提出G-Loss，一种基于图的损失函数，通过半监督标签传播机制利用嵌入流形内的结构关系。G-Loss构建文档相似性图以捕捉全局语义关联，引导模型学习更具区分性和鲁棒性的嵌入表示。我们在五个基准数据集上评估：MR（情感分析）、R8和R52（主题分类）、Ohsumed（医学文档分类）、20NG（新闻分类）。多数实验设置下，G-Loss收敛更快，生成更语义连贯的嵌入空间，分类准确率优于传统损失函数微调的模型。
原文摘要 · Abstract (English)
Traditional loss functions, including cross-entropy, contrastive, triplet, and su pervised contrastive losses, used for fine-tuning pre-trained language models such as BERT, operate only within local neighborhoods and fail to account for the global semantic structure. We present G-Loss, a graph-guided loss function that incorporates semi-supervised label propagation to use structural relationships within the embedding manifold. G-Loss builds a document-similarity graph that captures global semantic relationships, thereby guiding the model to learn more discriminative and robust embeddings. We evaluate G-Loss on five benchmark datasets covering key downstream classification tasks: MR (sentiment analysis), R8 and R52 (topic categorization), Ohsumed (medical document classification), and 20NG (news categorization). In the majority of experimental setups, G-Loss converges faster and produces semantically coherent embedding spaces, resulting in higher classification accuracy than models fine-tuned with traditional loss functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。