测试大模型是否理解事件时间流处理逻辑,发现表现不佳。
Can Large Language Models Reason about Event-Time Stream-Processing Semantics?
- 设计新基准测试,让模型模拟事件时间窗口处理
- 仅0.85的准确率接近满分,多数模型对迟到数据处理错误
- 适合研究流式系统与大模型交互的开发者参考
随着流处理系统越来越多地依赖大语言模型(LLMs)完成管道编写、告警分类和日志解析,一个关键假设是模型具备事件时间流处理的理解能力。本文提出 StreamReason-Bench 基准,要求模型在给定带窗口的查询和乱序事件流时,准确判断哪些窗口触发、聚合结果及哪些事件因迟到被丢弃。答案由符合 Dataflow 模型语义的小型参考实现生成,支持精确评分与部分分行 F1 评估,无需实际运行引擎。在涵盖滚动、滑动、会话窗口及处理时间窗口的600个生成样本上,模型在事件时间任务中表现差:直接回答时无模型达到34%以上精确匹配;链式思考(CoT)使多个模型准确率翻倍(如 GPT-4o 从 0.34 提升至 0.48),仅有单一前沿模型默认推理下接近解决(0.85)。处理时间控制组(无水印、无迟到)几乎被所有合格模型正确解答。差距主要来自事件时间和迟到数据处理,而非窗口机制或计算本身。按窗口类型分析排序错误显示,迟到数据错误主导事件时间窗口失败,而在控制组消失;会话窗口主要错误在于会话边界判定。
原文摘要 · Abstract (English)
Streaming systems increasingly hand work to large language models (LLMs): writing pipelines, triaging alerts, reading logs. All of it assumes the model knows how event-time stream processing behaves, and we test that assumption directly. StreamReason-Bench asks a model to stand in for an event-time stream processor. Given a windowed query and a stream of out-of-order events, it reports which windows fire, with their aggregates, and which events are dropped as late. The answer key comes from a small reference implementation of Dataflow-model semantics, so we can grade exactly, and with a partial-credit row-F1, without running an engine. On 600 generated items covering tumbling, hopping, session, and processing-time windows, the models do poorly on event time. When told to answer directly, no model that actually follows the instruction clears 34% exact match; chain-of-thought (CoT) roughly doubles that for several of them (GPT-4o goes from 0.34 to 0.48), and only one frontier model that reasons by default comes near solving the set (0.85). A processing-time control, with no watermarks and nothing late, is almost solved by every capable model. The gap points to event-time and late-data handling, not windowing or arithmetic, as the hard part. Sorting errors by window type tells the same story: late-data mistakes dominate the event-time windows and vanish on the control, while session windows mostly fail on where the session boundaries fall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。