无需真实标签,用多个错误模型预测结果纠正偏误,让AI数据可用。
Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements
- 利用多个不完美AI测量值,通过潜在真值建模消除误差。
- 即使准确率超90%,传统方法仍会引入偏误,而本方法可纠正。
- 适合社会科学研究者使用大模型标注数据但无真实标签的场景。
越来越多研究者使用AI测量变量并用于后续分析。尽管AI测量准确率可能高于90%,但忽略其预测误差会导致下游分析出现显著偏误和无效置信区间。现有方法如基于设计的监督学习或预测驱动推断需依赖代价高昂的真实标签。本文提出无真实标签的多不完美测量去偏推断框架(DMM),结合多个有误差的AI测量,假设其在潜在真值和单位特征(如文本嵌入)条件下独立。基于CP分解理论,证明该估计量一致且渐近正态,支持广泛的社会科学统计推断。模拟显示DMM可实现有效推断,加入更准确但非完美的测量能提升效率。针对大语言模型标注应用,还开发了检验条件独立性的诊断工具。
原文摘要 · Abstract (English)
An increasing number of scholars use AI to measure variables they subsequently include in downstream analyses. Although AI-measured variables are often analyzed as if observed without error, ignoring prediction errors in automated measurement leads to substantial bias and invalid confidence intervals in downstream analyses, even if AI measurement accuracy is high, e.g., above 90%. Existing solutions, such as design-based supervised learning and prediction-powered inference, combine error-prone AI-based measurements with gold-standard labels, which may be costly and difficult to obtain in some application areas. In this paper, we propose debiased inference with multiple imperfect measurements (DMM), a framework that combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels. Building on the established results on CP decomposition, DMM assumes that these measurements are independent conditional on the latent true label and observed unit-level features, such as text features represented by embeddings. This framework allows for unknown misclassification rates to vary across annotation methods (e.g., large language models) and across units of annotation (e.g., texts). Under this assumption, we use semiparametric inference theory to prove that the DMM estimator is consistent and asymptotically normal, enabling valid inference for a wide range of downstream statistical analyses common in the social sciences. Our simulation results show that DMM yields valid inference and that adding accurate, though imperfect, measurements can improve efficiency. Focusing on common applications of large language model annotations, we also develop diagnostics to assess the conditional independence assumption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。