用置信度指导专家与AI协作，提升土壤剖面标注效率
Uncertainty-Guided Expert-AI Collaboration for Efficient Soil Horizon Annotation
- 基于共形预测校准模型置信度，动态触发专家标注
- 相同标注预算下，回归任务效率提升，分类任务性能相当
- 适合资源有限的农业或环境领域智能标注场景
不确定性量化在人机协作中至关重要，因人类决策会随机器置信度调整。可靠校准的模型不确定性可实现更高效的协作、精准的专家干预和更负责任的机器学习系统使用。共形预测已成为一种广泛适用的模型无关框架，为回归和分类任务提供统计上有效的置信估计。本文将共形预测应用于多模态多任务土壤剖面描述模型SoilNet，设计了一个模拟人类在环（HIL）的标注流程，在模型不确定性高时才动用有限的专家标注预算。实验表明，对SoilNet进行共形化处理后，在相同标注预算下，回归任务的标注效率更高，分类任务的性能表现与非共形版本相当。所有代码与实验均可在项目仓库中获取：https://github.com/calgo-lab/BGR
原文摘要 · Abstract (English)
Uncertainty quantification is essential in human-machine collaboration, as human agents tend to adjust their decisions based on the confidence of the machine counterpart. Reliably calibrated model uncertainties, hence, enable more effective collaboration, targeted expert intervention and more responsible usage of Machine Learning (ML) systems. Conformal prediction has become a well established model-agnostic framework for uncertainty calibration of ML models, offering statistically valid confidence estimates for both regression and classification tasks. In this work, we apply conformal prediction to $\textit{SoilNet}$, a multimodal multitask model for describing soil profiles. We design a simulated human-in-the-loop (HIL) annotation pipeline, where a limited budget for obtaining ground truth annotations from domain experts is available when model uncertainty is high. Our experiments show that conformalizing SoilNet leads to more efficient annotation in regression tasks and comparable performance scores in classification tasks under the same annotation budget when tested against its non-conformal counterpart. All code and experiments can be found in our repository: https://github.com/calgo-lab/BGR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。