提出三数报告法,解决遥感分类器评估中先验失配导致的精度虚高问题。
Prior-matched evaluation of operational Earth-observation classifiers: a three-number reporting method demonstrated on Sentinel-1 internal-wave detection

- 基于真实数据流先验,用三个数字(平衡测试、操作先验、部署后)进行评估
- 实际运行精度从0.192提升至0.927,验证方法可有效缩小评估与现实差距
- 适合构建稀有事件检测系统的科研人员和工程团队参考
内部波服务从哨兵-1波模式档案中筛查内孤立波,将检测结果提交专家审核,而专家审阅时间是需要节约的核心资源。由于注意力成本来自错误,精度优先至关重要。该分类器在类平衡条件下训练并报告,但实际运行率约为每二十幅图像一例,平衡测试得分严重夸大了验证者面临的精度。一个在平衡测试中得分为0.794的模型,在实际运行中仅得0.192精度:这一差距是因在错误先验下报告所致,且被普遍使用的指标所掩盖。我们证明,此不匹配实为评估问题而非训练问题,在固定召回率下,先验校正与校准均无法提升精度。为此提出先验匹配报告方法,基于三个数值:平衡测试、操作先验、部署后真实表现,其差异即为真实衡量标准。通过精度优先、泄漏可控的开发流程,逐级优化模型,每一步仅在预注册阈值上通过;负样本多样性增强、聚合头提升,容量仅一次投入即停止,校准无效,因此真实负样本亦成为成果一部分。在召回率不低于0.80的前提下,使用密封单次读取锁箱认证,优化模型在操作先验下报告精度达0.927;跨时间检验确认判别能力可迁移至未见时段,而固定操作点则不行。先验匹配报告法——先以平衡条件开始,再随数据流揭示的先验逐步过渡——可推广至任何尚不知晓先验的地球观测服务,用于构建稀有事件探测器。
原文摘要 · Abstract (English)
The Internal Waves Service screens the Sentinel-1 Wave-mode archive for internal solitary waves, routing detections to experts whose adjudication time is the resource the effort exists to conserve. Because attention is the cost of error, precision leads. Its classifier was trained and reported at a one-to-one class balance, fixed before the operational rate could be known. That rate has since emerged at roughly one scene in twenty, and a balanced-test score badly overstates the precision a validator meets. A model that scores 0.794 balanced-test precision scores 0.192 in real operation: the gap is a systematic artefact of reporting at the wrong prior, invisible to the metric most work quotes. We show the mismatch to be an evaluation problem in the costume of a training one at a fixed recall, prior correction and calibration cannot move precision, and answer it with a prior-matched reporting method based on three numbers: balanced-test, operational-prior, and real post-deployment, whose contrast is the honest measure. A precision-first, leakage-controlled development cycle then improves the classifier lever by lever, each promoted only against a pre-registered margin; negative variety and the aggregation head lifting, capacity paying once then stopping, calibration inert, so the honest negatives are as much a result as the gains. Holding recall at a floor of 0.80 and certifying against a sealed, single-read lockbox, the promoted model reports 0.927 precision at the operational prior; an out-of-time check confirms discrimination transfers to unseen periods while a fixed operating point does not. Prior-matched reporting, begin balanced, then move to the prior as the stream reveals it, transfers to any operational Earth-observation service bootstrapping a rare-event detector under a prior it has yet to discover.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。