针对混合语言文本设计双解码器模型，提升跨语言切换点识别能力。
CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modeling
- 采用共享编码器与同步交叉注意力的双解码器架构，捕捉混合语言结构。
- 在HASOC-2021上实现更高的准确率、精确率和F1值，优于现有方法。
- 适合研究多语言混合文本的NLP工程师或社会媒体分析人员。
混合语言以句内频繁切换为特征，给标准语言模型带来结构挑战。本文提出CMLFormer，一种基于共享编码器与同步解码器交叉注意力的多层双解码器Transformer，用于建模混合语言的语义与语言动态。该模型在扩充的印地-英语（Hinglish）语料库上进行预训练，包含切换点与翻译标注，并引入多项新目标，专门捕捉切换行为、跨语言结构与混合复杂性。实验表明，在特定预训练设置下，CMLFormer在HASOC-2021基准测试中显著提升F1分数、精确率与准确率。注意力分析显示模型能有效识别并聚焦于切换点，验证其对混合语言结构的敏感性。结果表明，CMLFormer的架构与多任务预训练策略在建模混合语言方面具有显著有效性。
原文摘要 · Abstract (English)
Code-mixed languages, characterized by frequent within-sentence language transitions, present structural challenges that standard language models fail to address. In this work, we propose CMLFormer, an enhanced multi-layer dual-decoder Transformer with a shared encoder and synchronized decoder cross-attention, designed to model the linguistic and semantic dynamics of code-mixed text. CMLFormer is pre-trained on an augmented Hinglish corpus with switching point and translation annotations with multiple new objectives specifically aimed at capturing switching behavior, cross-lingual structure, and code-mixing complexity. Our experiments show that CMLFormer improves F1 score, precision, and accuracy over other approaches on the HASOC-2021 benchmark under select pre-training setups. Attention analyses further show that it can identify and attend to switching points, validating its sensitivity to code-mixed structure. These results demonstrate the effectiveness of CMLFormer's architecture and multi-task pre-training strategy for modeling code-mixed languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。