用最优采样策略在有限预算下精准标注社工记录,提升因果推断效率。
Optimal Causal Annotations: An Application to Casenotes in Social Services
- 基于方差最小化设计标注优先级,实现最优样本选择
- 真实数据实验显示标签量减少43%-91%而区间精度不变
- 揭示街头援助对住房进展的边际效应递减规律
估算干预措施的因果效应是政策与运营的核心,但结果数据常缺失或获取成本高。大语言模型(LLM)可规模化进行文本标注,但可能存在未知偏差。当真实结果需昂贵专家标注或随访时,预算限制通常只能标注部分数据。受一家非营利组织在无家可归者服务中街头外展工作的启发,其关键结果存在于非结构化案情记录中,我们提出:在固定预算下应如何选择观测样本进行标注?我们的方法优化标注概率以最小化平均处理效应估计的方差,推导出闭式解,并证明可行的两阶段估计器达到最优渐近方差。在模拟和真实数据集上,该方法均显著降低均方误差,相比随机采样,标签数与成本减少43%至91%且保持相同置信区间宽度。在一项将案情记录中的住房申请进展分类的任务中,发现现成的LLM低估了客户进展,凸显了对齐模型判断的重要性。估计表明,前六个月内增加街头外展使8.6%的客户两年住房结果改善,从第一到第三四分位数扩大外展范围,住房申请进展预计提升半步。管理启示:将LLM作为裁判可能威胁估计有效性;本方法可在有限标注预算下实现有效因果推断。评估中间性非营利结果的因果影响可指导运营决策:外展回报呈凹形,因此扩大覆盖人群的广度可能更具资源效益。
原文摘要 · Abstract (English)
Problem definition: Estimating causal effects of interventions is central to policy and operations, but outcome data are often missing or costly to obtain. LLMs can provide text annotation at scale but may be subject to unknown bias. When ground-truth outcomes require expensive expert labeling or follow-up, budget limits typically allow only a fraction of the data to be labeled. Motivated by collaboration with a nonprofit conducting street outreach in homelessness services, whose most interesting outcomes are in unstructured casenotes, we ask: which observations should be selected for labeling under a fixed budget? Methodology/results: Our method optimizes annotation probabilities to minimize the variance of average treatment effect estimation. We derive a closed-form solution and establish that a feasible two-batch estimator achieves the best possible asymptotic variance. On simulated and real-world datasets, our method achieves lower MSE than random sampling and 43%-91% reductions in labels and costs for the same interval widths. In a case study classifying progress towards housing applications from casenotes with LLMs, we find out-of-the-box LLMs under-recognize client progress, highlighting the importance of grounding LLM judgment. Our estimates indicate that 8.6% of clients improve 2-year housing outcomes due to more street outreach in the first six months, and increasing outreach from the first to third quartile increases maximum progress towards a housing application by an estimated half-step. Managerial implications: LLM-as-a-judge may threaten estimation validity; our method enables valid causal estimation with a limited annotation budget. Evaluating causal impacts on intermediate nonprofit outcomes can inform operations: returns to outreach appear concave, so expanding the extensive margin of who receives outreach may be resource-efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。