构建真实物流客服语料库,揭示翻译数据对多语言意图识别的高估问题。
From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service
- 基于真实客服日志构建600K条过滤后数据,含30K独立查询。
- 翻译数据集性能比原生数据高15%-20%,尤其在长尾意图上偏差显著。
- 支持零样本跨语言评估,适合多语言NLP系统真实场景测试。
多语言意图分类是全球物流平台客服系统的核心,模型需处理跨语言、带噪声的用户请求及层级标签体系。现有多数多语言基准依赖机器翻译文本,通常比真实客户请求更干净、标准化,导致对实际鲁棒性评估过高。本文提出一个公开的层次化多语言意图分类基准,数据源自真实物流客服日志。经筛选、LLM辅助质量控制与人工验证,最终获得约30,000条去标识化独立查询,来自600,000条历史记录,按两级分类体系组织,包含13个父级意图和17个子级意图。涵盖英语、西班牙语、阿拉伯语(可见语言),以及印尼语、中文及额外仅用于测试的语言,支持零样本评估。为直接衡量合成与真实评估间的差距,提供成对的原生与翻译测试集,并在平铺与层次协议下评测多种多语言编码器、嵌入模型及小语言模型。结果表明,翻译测试集在噪声原生查询上的表现显著高估,尤其在长尾意图和跨语言迁移任务中,凸显构建更真实多语言基准的必要性。
原文摘要 · Abstract (English)
Multilingual intent classification is central to customer-service systems on global logistics platforms, where models must process noisy user queries across languages and hierarchical label spaces. Yet most existing multilingual benchmarks rely on machine-translated text, which is typically cleaner and more standardized than native customer requests and can therefore overestimate real-world robustness. We present a public benchmark for hierarchical multilingual intent classification constructed from real logistics customer-service logs. The dataset contains approximately 30K de-identified, stand-alone user queries curated from 600K historical records through filtering, LLM-assisted quality control, and human verification, and is organized into a two-level taxonomy with 13 parent and 17 leaf intents. English, Spanish, and Arabic are included as seen languages, while Indonesian, Chinese, and additional test-only languages support zero-shot evaluation. To directly measure the gap between synthetic and real evaluation, we provide paired native and machine-translated test sets and benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical protocols. Results show that translated test sets substantially overestimate performance on noisy native queries, especially for long-tail intents and cross-lingual transfer, underscoring the need for more realistic multilingual intent benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。