轻量级模型提升跨领域语言识别准确率，尤其适合语音助手等低延迟场景。
PolyLingua: Margin-based Inter-class Transformer for Robust Cross-domain Language Detection
- 采用两级对比学习框架，动态调整类间距离，增强嵌入区分度。
- 在亚马逊和歌曲数据集上分别达到99.25%和98.15%的F1值。
- 参数量仅为Sonnet 3.5的十分之一，适合资源受限环境。
语言识别是多语言系统（如聊天机器人、虚拟助手）的关键前置步骤，错误会引发下游任务失败。现有工具在音乐请求等场景表现不佳——用户语种与歌名语言常不一致。开源工具如LangDetect、FastText虽快但准确率低，大模型虽有效却难以用于低延迟或低资源环境。本文提出PolyLingua，一种基于轻量Transformer的域内语言识别与细粒度分类模型。其采用两层对比学习：实例级分离与类级对齐，结合自适应边缘机制，生成紧凑且分离良好的嵌入表示，即使对相似语言也具强区分能力。在两个挑战性数据集上评估：Amazon Massive（多语言语音助手语句）与Song数据集（频繁混码的音乐请求），PolyLingua分别取得99.25%和98.15%的F1值，超越Sonnet 3.5，同时仅使用其十分之一参数量，适用于计算与延迟受限场景。
原文摘要 · Abstract (English)
Language identification is a crucial first step in multilingual systems such as chatbots and virtual assistants, enabling linguistically and culturally accurate user experiences. Errors at this stage can cascade into downstream failures, setting a high bar for accuracy. Yet, existing language identification tools struggle with key cases -- such as music requests where the song title and user language differ. Open-source tools like LangDetect, FastText are fast but less accurate, while large language models, though effective, are often too costly for low-latency or low-resource settings. We introduce PolyLingua, a lightweight Transformer-based model for in-domain language detection and fine-grained language classification. It employs a two-level contrastive learning framework combining instance-level separation and class-level alignment with adaptive margins, yielding compact and well-separated embeddings even for closely related languages. Evaluated on two challenging datasets -- Amazon Massive (multilingual digital assistant utterances) and a Song dataset (music requests with frequent code-switching) -- PolyLingua achieves 99.25% F1 and 98.15% F1, respectively, surpassing Sonnet 3.5 while using 10x fewer parameters, making it ideal for compute- and latency-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。