跨语言间接问答研究揭示高/低资源语言均存在识别难题
Indirect Question Answering in English, German and Bavarian: A Challenging Task for High- and Low-Resource Languages Alike
- 构建英德巴伐利亚三语间接问答数据集，含人工标注与GPT生成数据
- 多语言模型在英德巴伐利亚上表现均差，英语仍低于60%准确率
- 提示生成数据质量不足，需更多真实语料提升间接语义理解能力
间接性是日常交流的常见特征，但在高资源与低资源语言的自然语言处理研究中均未得到充分探索。间接问答（IQA）旨在分类间接回答的极性。本文提出两个覆盖英语、标准德语和巴伐利亚语（无标准拼写的德语方言）的多语言IQA语料库：InQA+为小规模高质量评估集，经人工标注；GenIQA为大规模训练集，包含由GPT-4o-mini生成的人工数据。基于多语言Transformer模型（mBERT、XLM-R、mDeBERTa）的多种实验表明，IQA是具有语用挑战性的任务。结果发现，即使在英语中性能也较低，且存在严重过拟合。分析显示标签模糊性、标签集设计和数据量是关键影响因素。无论在高资源语言（英语、德语）还是低资源语言（巴伐利亚语），性能均不佳，但大量训练数据有益。此外，GPT-4o-mini在各测试语言中均无法生成高质量IQA数据。
原文摘要 · Abstract (English)
Indirectness is a common feature of daily communication, yet is underexplored in NLP research for both low-resource as well as high-resource languages. Indirect Question Answering (IQA) aims at classifying the polarity of indirect answers. In this paper, we present two multilingual corpora for IQA of varying quality that both cover English, Standard German and Bavarian, a German dialect without standard orthography: InQA+, a small high-quality evaluation dataset with hand-annotated labels, and GenIQA, a larger training dataset, that contains artificial data generated by GPT-4o-mini. We find that IQA is a pragmatically hard task that comes with various challenges, based on several experiment variations with multilingual transformer models (mBERT, XLM-R and mDeBERTa). We suggest and employ recommendations to tackle these challenges. Our results reveal low performance, even for English, and severe overfitting. We analyse various factors that influence these results, including label ambiguity, label set and dataset size. We find that the IQA performance is poor in high- (English, German) and low-resource languages (Bavarian) and that it is beneficial to have a large amount of training data. Further, GPT-4o-mini does not possess enough pragmatic understanding to generate high-quality IQA data in any of our tested languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。