用自监督学习构建水利流量质检大模型,自动识别并修复海量传感器数据异常。
HydroGEM: A Self Supervised Zero Shot Hybrid TCN Transformer Foundation Model for Continental Scale Streamflow Quality Control
- 基于603万条清洁数据预训练,融合TCN与Transformer捕捉长短时依赖。
- 检测F1达0.792,重建误差降低68.7%,优于最强基线36.3%。
- 跨国家适用性强,适合需要自动化水文数据清洗的科研与水利部门。
传感器网络的发展实现了实时流量监测,但传感器故障持续影响数据质量。人工质检难以应对每年数百万条数据的规模。我们提出HydroGEM,一种用于大陆尺度流量质量控制的基础模型,旨在辅助专家工作。该模型在3,724个美国地质调查局(USGS)站点的603万条清洁序列上进行自监督预训练,学习通用水文特征,随后通过合成异常数据微调以实现检测与重建。采用混合TCN-Transformer架构(1420万参数),有效捕捉局部与长程时间依赖;分层归一化处理跨度达六数量级的流量变化。在799个预留站点、18种基于USGS标准的合成异常测试中,检测F1为0.792,重建误差降低68.7%,优于最强基线36.3%。在加拿大环境与气候变化部100个站点的跨国验证中,容忍评估(±24小时缓冲)下取得Tolerant F1=0.70,段级事件检测率达90.1%,展示良好跨区域泛化能力。模型在不同修正幅度下表现稳定,且峰值标记出现在冬季冰期,与水文学家实际修正行为一致。训练异常简化而测试异常复杂的设计,证明性能源于对水文规律的学习而非记忆。
原文摘要 · Abstract (English)
Advances in sensor networks have enabled real-time stream discharge monitoring, yet persistent sensor malfunctions limit data utility. Manual quality control by expert hydrologists cannot scale with networks generating millions of measurements annually. We introduce HydroGEM, a foundation model for continental-scale streamflow quality control designed to support human expertise. HydroGEM uses self-supervised pretraining on 6.03 million clean sequences from 3,724 USGS stations to learn general hydrological representations, followed by fine-tuning with synthetic anomalies for detection and reconstruction. A hybrid TCN-Transformer architecture (14.2M parameters) captures both local and long-range temporal dependencies, while hierarchical normalization handles six orders of magnitude in discharge. On held-out observations from 799 stations with 18 synthetic anomaly types grounded in USGS standards, HydroGEM achieves F1=0.792 for detection and 68.7% reconstruction error reduction, outperforming the strongest baseline by 36.3%. For cross-national validation on 100 Environment and Climate Change Canada stations using tolerant evaluation with a plus or minus 24-hour buffer, HydroGEM achieves Tolerant F1=0.70 with 90.1% segment-level event detection, demonstrating cross-national generalization. The model maintains consistent detection across correction magnitudes and aligns with operational seasonal patterns, with peak flagging during winter ice-affected periods matching hydrologists' correction behavior. Architectural separation between simplified training anomalies and complex test anomalies confirms that performance reflects learned hydrometric principles rather than pattern memorization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。