arXiv:2506.21445cs.CLcs.IR2025-06被引 3

跨语言文本转Cypher查询,发现多语训练能缩小性能差距

Text2Cypher Across Languages: Evaluating and Finetuning LLMs

  • 构建中英土三语数据集,评估大模型跨语言生成能力
  • 英文表现最佳,土耳其语最差,主因训练数据和语言差异
  • 多语言微调可平衡各语种表现,提升系统包容性

大语言模型已实现自然语言到数据库查询的转换,如Text2SQL、Text2SPARQL和Text2Cypher。然而当前研究多集中于英语,缺乏对其他语言的评估。本文在多种语言上评估基础与微调模型在Text2Cypher任务中的表现,通过翻译英文问题至西班牙语和土耳其语并保留原始Cypher查询,构建了多语言数据集。采用标准提示和指标,发现模型性能呈一致规律:英语最优,西班牙语次之,土耳其语最低,归因于训练数据量及语言特性差异。测试提示词翻译的影响,结果显示指标变化极小,说明提示翻译影响有限。进一步在英语单语和多语言数据集上微调模型,结果表明英语微调虽提升整体准确率但加剧语言间差距,而多语言微调则缩小差距,实现更均衡的表现。研究强调多语言评估与训练对构建更具包容性和鲁棒性的查询生成系统至关重要。

原文摘要 · Abstract (English)

Recent advances in large language models (LLMs) have enabled natural language interfaces that translate user questions into database queries, such as Text2SQL, Text2SPARQL, and Text2Cypher. While these interfaces enhance database accessibility, most research today focuses on English, with limited evaluation in other languages. This paper investigates the performance of both foundational and finetuned LLMs on the Text2Cypher task across multiple languages. We create and release a multilingual dataset by translating English questions into Spanish and Turkish while preserving the original Cypher queries, enabling fair cross-lingual comparison. Using standardized prompts and metrics, we evaluate several foundational models and observe a consistent performance pattern: highest on English, followed by Spanish, and lowest on Turkish. We attribute this to differences in training data availability and linguistic features. We also examine the impact of translating task prompts into Spanish and Turkish. Results show little to no change in evaluation metrics, suggesting prompt translation has minor impact. Furthermore, we finetune a foundational model on two datasets: one in English only, and one multilingual. Finetuning on English improves overall accuracy but widens the performance gap between languages. In contrast, multilingual finetuning narrows the gap, resulting in more balanced performance. Our findings highlight the importance for multilingual evaluation and training to build more inclusive and robust query generation systems.

文本转查询多语言Cypher微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。