用整数规划改进多语言混杂文本识别，提升准确率
Improving Language Identification for Code-Switched Utterances with Integer Linear Programming
- 将语言识别重构为整数线性规划问题，引入可解释约束
- 在10种语言的混合语料上性能显著提升
- 无需训练，适合低资源多语言场景使用
代码切换（CS）话语的语言识别仍是语言识别系统的一大挑战，导致这类文本在大型语言模型的训练数据中占比不足。本文重新审视了当前最先进的无训练方法MaskLID，该方法可检测任意语言组合。我们提出三大贡献：(a) 揭示并解决MaskLID对词级语言关联分数的过度依赖；(b) 将底层优化算法重构为整数线性规划，支持大量清晰且可解释的约束实验；(c) 两项改进均显著提升基线系统性能，在涵盖10种不同语言的实验中，对多种代码切换基准测试均有明显提升。代码与数据已公开以保障可复现性。
原文摘要 · Abstract (English)
Automatic identification of code-switched (CS) utterances remains a challenge for language identification (LID) systems, causing such texts to be underrepresented in the training data of Large Language Models. In this paper, we revisit MaskLID, a state-of-the art approach for CS identification, which requires no training and detects arbitrary language combinations. We make three main contributions: (a) we reveal, and address, a major issue of MaskLID: its overreliance on word-level language association scores; (b) we reformulate the underlying optimization algorithm as an Integer Linear Program, enabling us to experiment with a large set of clear and interpretable constraints; (c) each of these improvements vastly improves the baseline system, as we illustrate in experiments involving 10~diverse languages, where we observe a strong boost in performance on CS benchmarks. We release our code and data for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。