比较三种跨设施数据流架构，发现直接传输最快，代理传输最易部署。
From Edge to HPC: Investigating Cross-Facility Data Streaming Architectures
- 提出三种数据流架构：直连、代理和托管服务，对比其路径与部署特性。
- 直连架构吞吐更高、延迟更低，广播/归约模式下性能领先30%以上。
- 适合需要低延迟的科研计算，尤其关注AI-HPC融合场景的工程师。
本文研究了三种跨设施数据流架构：直连流（DTS）、代理流（PRS）和托管服务流（MSS）。通过在橡树岭领导力计算设施（OLCF）的生产级先进计算生态（ACE）基础设施上，采用数据流到高性能计算（DS2HPC）框架与SciStream内存到内存流工具包实现，评估其数据流路径与部署可行性。基于三类源自科学工作流特征的合成负载，模拟实验测定了在工作共享、带反馈的工作共享、广播与聚集等常见通信模式下的吞吐量、往返时间与开销。结果表明，DTS具有最少跳数，带来更高吞吐与更低延迟；MSS虽部署更灵活、支持多用户扩展，但开销显著；PRS介于两者之间，在多数情况下性能接近DTS。
原文摘要 · Abstract (English)
In this paper, we investigate three cross-facility data streaming architectures, Direct Streaming (DTS), Proxied Streaming (PRS), and Managed Service Streaming (MSS). We examine their architectural variations in data flow paths and deployment feasibility, and detail their implementation using the Data Streaming to HPC (DS2HPC) architectural framework and the SciStream memory-to-memory streaming toolkit on the production-grade Advanced Computing Ecosystem (ACE) infrastructure at Oak Ridge Leadership Computing Facility (OLCF). We present a workflow-specific evaluation of these architectures using three synthetic workloads derived from the streaming characteristics of scientific workflows. Through simulated experiments, we measure streaming throughput, round-trip time, and overhead under work sharing, work sharing with feedback, and broadcast and gather messaging patterns commonly found in AI-HPC communication motifs. Our study shows that DTS offers a minimal-hop path, resulting in higher throughput and lower latency, whereas MSS provides greater deployment feasibility and scalability across multiple users but incurs significant overhead. PRS lies in between, offering a scalable architecture whose performance matches DTS in most cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。