用因果模型分析代码生成中不同提示的影响。
CodeSCM: Causal Analysis for Multi-Modal Code Generation
- 构建因果模型分离提示中代码与自然语言语义
- 发现输入输出样例显著影响代码生成结果
- 适合研究大模型生成机制的学者参考
本文提出CodeSCM，一种用于分析大语言模型在多模态代码生成中的结构因果模型（SCM）。通过在CodeSCM上施加干预，我们度量了自然语言、代码及输入输出示例等不同提示模态对模型的影响。CodeSCM引入隐变量以分离多模态提示中代码与自然语言的语义。基于因果中介分析原理，量化了代表模型虚假倾向的直接效应。结果显示，除自然语言指令外，输入输出示例对代码生成具有显著影响。
原文摘要 · Abstract (English)
In this paper, we propose CodeSCM, a Structural Causal Model (SCM) for analyzing multi-modal code generation using large language models (LLMs). By applying interventions to CodeSCM, we measure the causal effects of different prompt modalities, such as natural language, code, and input-output examples, on the model. CodeSCM introduces latent mediator variables to separate the code and natural language semantics of a multi-modal code generation prompt. Using the principles of Causal Mediation Analysis on these mediators we quantify direct effects representing the model's spurious leanings. We find that, in addition to natural language instructions, input-output examples significantly influence code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。