用新合成语言提升模型向英语迁移能力，效果优于已有方法。
Transfer of Structural Knowledge from Synthetic Languages
- 用新合成语言微调模型，增强结构知识迁移。
- 在多个任务上表现更好，尤其对弱模型更有效。
- 引入小规模评测基准Tiny-Cloze，适合评估轻量模型。
本文研究将多种合成语言的结构知识迁移至英语。通过分析微调后模型的嵌入结构、所含信息及其在简单语言任务上的能力，发现新合成语言相比以往研究中的语言能带来更好的迁移效果。为此，我们提出一种新合成语言，显著提升对英语的迁移性能。同时引入小型封闭式填空评测基准Tiny-Cloze，用于评估不同领域微调模型的自然语言理解能力。实验表明，在新合成语言上微调可显著提升模型在多任务上的表现，尤其对计算资源受限的模型更具优势。
原文摘要 · Abstract (English)
This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark - a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。