为凯克天文台档案开发实时数据看板,提升运维效率与可扩展性。
An Interactive Metrics Dashboard for the Keck Observatory Archive
- 基于Python和Plotly-Dash构建新型查询系统,支持近实时数据接入。
- 查询速度提升20倍,实现分钟级数据可用与实时性能监控。
- 适合天文数据管理者、系统架构师及需要高时效性分析的科研人员。
自2004年以来,凯克天文台档案(KOA)由美国宇航局(NASA)资助,由美国系外行星科学研究所(NExScI)与W.M.凯克天文台合作运营,负责接收并分发夏威夷莫纳克亚山双10米凯克望远镜的所有观测数据。过去三年,KOA启动现代化改造,用基于Python的新架构替代原有系统,以应对新仪器产生的大量复杂数据,并支持对维拉C·鲁宾天文台等巡天望远镜产生的海量瞬变源警报进行后续追踪。自2022年起,KOA已实现近实时数据摄入,通常在数据生成后一分钟内完成处理并可通过专用网页界面访问。当前部署的新查询基础设施采用Plotly-Dash框架与R树索引,使查询速度提升20倍。本文利用该新基础设施开发了一个集成仪表板,用于实时获取档案性能与增长的关键指标,评估系统健康状况,并指导未来软硬件升级规划。原有度量方法因延迟高、不支持按需查询、分散于多个工具且使用繁琐,已无法满足持续扩展的需求。
原文摘要 · Abstract (English)
Since 2004, the Keck Observatory Archive (KOA) has operated as a NASA-funded collaboration between the NASA Exoplanet Science Institute ( NExScI) and the W.M. Keck Observatory. It ingests and serves all data acquired by the twin 10-meter Keck telescopes on Mauna Kea, Hawaii. In the past three years, KOA has begun a modernization program to replace the architecture and systems used since the archive's creation with a new modern Python-based infrastructure. This infrastructure will position KOA to respond to the rapid growth of new and complex data sets that will be acquired by new instruments now in development, and enable follow-up to identify the deluge of alerts of transient sources expected by new survey telescopes such as the Vera C. Rubin Observatory. Since 2022, KOA has ingested new data in near-real time, generally within one minute of creation, and has made them immediately accessible to observers through a dedicated web interface. The archive is now deploying a new, scalable, Python-based, VO-compliant query infrastructure built with the Plotly-Dash framework and R-tree indices to speed-up queries by a factor of 20. The project described here exploits the new query infrastructure to develop a dashboard that will return live metrics on the performance and growth of the archive. These metrics assess the current health of the archive and guide planning future hardware and software upgrades. This single dashboard will enable, for example, monitoring of real-time ingestion, as well as studying the long-term growth of the archive. Current methods of gathering metrics that have been in place since the archive opened will not support the archive as it continues to scale. These methods suffer from high latency, are not optimized for on-demand metrics, are scattered among various tools, and are cumbersome to use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。