arXiv:2512.02053cs.LG2025-12

将结构特征注入Transformer中间层,提升歌词分类准确率与概率可靠性

Contextual Gating within the Transformer Stack: Synergistic Feature Modulation for Enhanced Lyrical Classification and Calibration

  • 在BERT编码器中间层引入上下文门控机制,动态调制语义特征
  • 在降维后歌词嵌入上实现99.10%准确率与F1,优于前代模型
  • 兼具高判别力与可信概率输出,适合需要可靠置信度的文本任务

本研究提出一种针对歌词内容分类的特征融合架构改进,将辅助结构特征直接融入预训练Transformer的自注意力机制。提出SFL Transformer模型,通过上下文门控机制(中间层SFL)调制BERT编码器堆栈中的隐藏状态序列,而非在输出层融合特征。该方法利用低维结构线索(Fstruct)调制深层语义特征(Hseq)。模型应用于从UMAP降维后的歌词嵌入中提取的二分类任务。SFL Transformer达到99.10%准确率和99.10%宏平均F1,显著优于先前发表的SFL模型(准确率0.9894)。关键的是,该策略保持极高可靠性,预期校准误差(ECE)仅为0.0081,对数损失为0.0489。结果验证了在中间层注入辅助上下文是协同融合结构与语义信息的最优方式,构建出兼具强判别能力与高保真概率估计的模型。

原文摘要 · Abstract (English)

This study introduces a significant architectural advancement in feature fusion for lyrical content classification by integrating auxiliary structural features directly into the self-attention mechanism of a pre-trained Transformer. I propose the SFL Transformer, a novel deep learning model that utilizes a Contextual Gating mechanism (an Intermediate SFL) to modulate the sequence of hidden states within the BERT encoder stack, rather than fusing features at the final output layer. This approach modulates the deep, contextualized semantic features (Hseq) using low-dimensional structural cues (Fstruct). The model is applied to a challenging binary classification task derived from UMAP-reduced lyrical embeddings. The SFL Transformer achieved an Accuracy of 0.9910 and a Macro F1 score of 0.9910, significantly improving the state-of-the-art established by the previously published SFL model (Accuracy 0.9894). Crucially, this Contextual Gating strategy maintained exceptional reliability, with a low Expected Calibration Error (ECE = 0.0081) and Log Loss (0.0489). This work validates the hypothesis that injecting auxiliary context mid-stack is the most effective means of synergistically combining structural and semantic information, creating a model with both superior discriminative power and high-fidelity probability estimates.

Transformer歌词分类特征融合概率校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。