arXiv:2604.21549cs.AIstat.ME2026-04被引 1

用多校准大模型更准估算类别真实比例,尤其在数据分布变化时。

Unbiased Prevalence Estimation with Multicalibrated LLMs

  • 基于输入特征的多校准方法,避免因数据分布偏移导致的估计偏差。
  • 实验显示标准方法偏差随分布偏移增大而上升,多校准方法始终接近零偏差。
  • 适用于大模型、诊断测试等场景,尤其适合跨区域或跨群体的流行率估算。

利用不完美的测量工具(如诊断测试、分类器或大语言模型)估算人群中某类别的真实比例,是科学、公共卫生及网络信任安全中的基础问题。传统方法依赖已知的设备误差率,但假设这些误差率在不同人群间保持不变。我们发现,在协变量偏移下该假设失效,而多校准(multicalibration)——即在输入特征条件下而非仅平均意义上实现校准——足以保证在偏移下的无偏估计。标准校准与量化方法无法提供此保障。我们的工作将公平性领域的最新理论与长期存在的测量问题相连接。模拟结果表明,标准方法的偏差随偏移程度增加而增长,而多校准估计器保持近似零偏差。尽管重点讨论大语言模型,但理论适用于任何分类模型。两个实证应用——使用美国社区调查估算各州就业率,以及用大语言模型分类四个国家的政治文本——证明多校准显著降低实际偏差,同时强调校准数据应覆盖目标群体可能差异的关键特征维度。

原文摘要 · Abstract (English)

Estimating the prevalence of a category in a population using imperfect measurement devices (diagnostic tests, classifiers, or large language models) is fundamental to science, public health, and online trust and safety. Standard approaches correct for known device error rates but assume these rates remain stable across populations. We show this assumption fails under covariate shift and that multicalibration, which enforces calibration conditional on the input features rather than just on average, is sufficient for unbiased prevalence estimation under such shift. Standard calibration and quantification methods fail to provide this guarantee. Our work connects recent theoretical work on fairness to a longstanding measurement problem spanning nearly all academic disciplines. A simulation confirms that standard methods exhibit bias growing with shift magnitude, while a multicalibrated estimator maintains near-zero bias. While we focus the discussion mostly on LLMs, our theoretical results apply to any classification model. Two empirical applications -- estimating employment prevalence across U.S. states using the American Community Survey, and classifying political texts across four countries using an LLM -- demonstrate that multicalibration substantially reduces bias in practice, while highlighting that calibration data should cover the key feature dimensions along which target populations may differ.

大模型校准流行率估计偏移鲁棒

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。