代码预训练提升结构化任务表现，但可能损害语言敏感性任务
How Does Code Pretraining Affect Language Model Task Performance?
- 通过控制数据量，对比语言与代码混合比例对模型影响
- 代码比例越高，语法解析和数学任务表现越好，但语言结构任务下降
- 适合关注代码增强对模型泛化能力影响的研究者
大型语言模型越来越多地在包含自然语言和源代码的语料上进行预训练。除了提升编程相关任务表现外，有迹象表明代码预训练可能改善其他无关任务的表现，但尚无研究能通过控制语言与代码数据比例建立因果关系。本文在此方面进行探索：在两种设置下预训练语言模型——加法型（总数据量恒定）与竞争型（语言数据量恒定），考察混合比例对性能的影响。实验涵盖BigBench基准中多样化任务，以及通过语义解析和句法变换衡量的组合性。结果表明，更高比例的代码预训练可提升涉及结构化输出（如语义解析）和数学任务的表现；反之，会损害对语言结构敏感的任务（如句法、形态）及现实知识类任务的表现。
原文摘要 · Abstract (English)
Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks, yet to date no work has been able to establish a causal connection by controlling between language and code data. Here we do just this. We pretrain language models on datasets which interleave natural language and code in two different settings: additive, in which the total volume of data seen during pretraining is held constant; and competitive, in which the volume of language data is held constant. We study how the pretraining mixture affects performance on (a) a diverse collection of tasks included in the BigBench benchmark, and (b) compositionality, measured by generalization accuracy on semantic parsing and syntactic transformations. We find that pretraining on higher proportions of code improves performance on compositional tasks involving structured output (like semantic parsing), and mathematics. Conversely, increase code mixture can harm performance on other tasks, including on tasks that requires sensitivity to linguistic structure such as syntax or morphology, and tasks measuring real-world knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。