用AI重写作文,拆解成绩差距中内容与表达的贡献
Content vs. Form: What Drives the Writing Score Gap Across Socioeconomic Backgrounds? A Generated Panel Approach
- 用大模型生成同一作文的多种表达版本,分离内容与风格影响
- 六分制下低收入学生作文平均差0.67分,内容差异占69%
- 方法可推广至教育评估研究,揭示隐性偏见来源
不同社会经济背景的学生在考试成绩上存在持续差距,这些差距可能转化为日后教育和劳动力市场的不平等。在许多评估中,表现不仅反映学生掌握的知识,还体现其表达知识的能力。这一区别在写作评估中尤为明显,评分同时考量思想内容与表达方式。因此,观察到的成绩差距可能混杂了内容差异与表达能力的差异。核心问题是:社会经济地位(SES)带来的成绩差距中有多少源于学生说什么,又有多少源于他们怎么说?我们基于美国中小学生撰写的大规模议论文语料库进行研究,提出一种新测量策略:利用大语言模型生成每篇作文的多个风格变体,保留原有论点但系统改变表达形式,构建“生成面板”,实现同一作文内部风格的可控变化。该方法使我们能够将写作成绩差距分解为内容与风格的贡献。结果发现,在1-6分量表上存在0.67分的SES差距,其中约69%归因于内容质量差异,风格差异占26%,评分标准差异占5%。这些模式在不同人口群体和写作任务中保持稳定。更广泛地,本方法展示了大语言模型如何在观测数据中生成可控变异,使研究人员能够分离并量化原本纠缠的因素。
原文摘要 · Abstract (English)
Students from different socioeconomic backgrounds exhibit persistent gaps in test scores, gaps that can translate into unequal educational and labor-market outcomes later in life. In many assessments, performance reflects not only what students know, but also how effectively they can communicate that knowledge. This distinction is especially salient in writing assessments, where scores jointly reward the substance of students' ideas and the way those ideas are expressed. As a result, observed score gaps may conflate differences in underlying content with differences in expressive skill. A central question, therefore, is how much of the socioeconomic-status (SES) gap in scores is driven by differences in what students say versus how they say it. We study this question using a large corpus of persuasive essays written by U.S. middle- and high-school students. We introduce a new measurement strategy that separates content from style by leveraging large language models to generate multiple stylistic variants of each essay. These rewrites preserve the underlying arguments while systematically altering surface expression, creating a "generated panel" that introduces controlled within-essay variation in style. This approach allows us to decompose SES gaps in writing scores into contributions from content and style. We find an SES gap of 0.67 points on a 1-6 scale. Approximately 69% of the gap is attributable to differences in essay content quality, Style differences account for 26% of the gap, and differences in evaluation standards across SES groups account for the remaining 5%. These patterns seems stable across demographic subgroups and writing tasks. More broadly, our approach shows how large language models can be used to generate controlled variation in observational data, enabling researchers to isolate and quantify the contributions of otherwise entangled factors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。