STREAM让研究者无缝切换本地、机构超算和云端大模型，实现低延迟交互。
STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming
- 三层次路由架构+本地复杂度判断，智能分配计算资源。
- 超算通过双通道流式传输，平均首令牌生成时间仅0.54秒。
- 支持无门槛调用超算，适合注重隐私与效率的研究团队。
研究人员在使用大语言模型时面临资源碎片化问题：本地模型免费私密但受限于硬件；机构超算中心提供强大算力且数据安全，但仅支持批处理作业；商业云接口虽可即时获取前沿模型，却存在高成本和数据留存限制。现有系统无法统一三者。STREAM（智能分层路由引擎）提出四项创新：(1) 结合本地、超算与云端的三层路由架构，辅以本地LLM进行复杂度判断；(2) 双通道超算流式架构，将控制平面（认证与任务调度）与数据平面（令牌传输）分离，通过WebSocket中继实现在不改防火墙策略或无需VPN的前提下，跨机构完成亚秒级首令牌生成时间（中位数0.54秒，较批处理模式提升21.1倍，原为11.40秒），端到端采用AES-256-GCM加密，确保中继方无法读取内容；(3) 分层感知的上下文摘要机制，避免长对话强制低复杂度请求占用高成本层级；(4) 超算即服务代理模式，将超算推理封装为兼容OpenAI接口的调用端点，任何标准客户端均可无感接入，其可行性依赖于第(2)项带来的亚秒级延迟。在涵盖十个领域的1,200个查询基准上，Llama 3.2 3B模型实现85.1%的免费层保留率。实测首令牌生成时间：本地0.26秒，超算（中继）0.54秒，云1.68秒。
原文摘要 · Abstract (English)
Researchers and practitioners working with large language models face a fragmented landscape: local models are free and private but hardware limits the model size and context windows a researcher can use; institutional HPC centers offer powerful GPU resources at no marginal cost and keep data within institutional boundaries, but operate behind firewalls and are designed for batch jobs rather than interactive use; commercial cloud APIs provide frontier-model quality on demand but impose significant cost and data retention policies unsuitable for sensitive research data. No existing system unifies all three. STREAM (Smart Tiered Routing Engine for AI Models) addresses this gap with four contributions: (1) a three-tier routing architecture combining local, HPC, and cloud inference with a local LLM-based complexity judge; (2) a dual-channel HPC streaming architecture that separates the Globus Compute control plane (authentication and job dispatch) from a WebSocket relay data plane (token delivery), enabling sub-second TTFT (0.54 s median, 21.1x over batch mode's 11.40 s) through institutional firewalls without VPN or firewall rule changes, with end-to-end AES-256-GCM encryption ensuring the relay operator cannot read token payloads; (3) tier-aware context summarization that prevents long conversations from forcing simple queries onto expensive tiers; and (4) an HPC-as-API proxy mode that exposes HPC inference as an OpenAI-compatible endpoint callable from any standard client with no HPC expertise, a deployment pattern made practical only by the sub-second TTFT of contribution (2). Llama 3.2 3B achieves 85.1% free-tier retention on a 1,200-query benchmark spanning ten domains. Measured TTFT: 0.26 s local, 0.54 s HPC (relay), 1.68 s cloud.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。