arXiv:2607.20638cs.AIcs.AR2026-07

评测大模型对数字波形的时序推理能力,发现复杂问题表现不佳。

WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms

论文配图:WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
图 1 · 摘自论文原文
  • 构建波形问答基准,含360个带程序化答案的问题。
  • 模型在复杂时序题上准确率下降,受限于上下文长度与推理能力。
  • 事件时间JSON格式比VCD格式更利于模型理解波形。

大型语言模型在代码生成和推理方面表现出色,但在数字波形数据上的时序推理能力尚未被充分探索。尽管波形推理是设计验证的关键瓶颈,现有基准主要评估硬件描述语言(HDL)代码生成,仅将波形作为辅助信息。本文提出WaveformQA,一个开源的波形问答基准,用于评估大模型在数字波形上的时序推理能力。该基准包含360个问题,覆盖八个难度等级,涵盖多信号相关性与事件顺序等类型,所有波形均来自开源设计实现,确保可复现性并基于真实硬件行为。对前沿大模型的评估显示,模型在简单查询上表现合理,但面对复杂时序与多步推理任务时性能下降,主要受制于上下文窗口限制和推理困难。此外,我们证明事件时间的JSON表示相比标准的值变化转储(VCD)格式能显著提升模型推理准确率。该开源框架支持扩展新问题类别与导入新波形源,便于研究者快速原型化时序推理实验。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated strong capabilities in code generation and reasoning, yet their ability to perform temporal reasoning over digital waveform data remains largely unexplored. Although reasoning over digital waveforms is a critical bottleneck in design verification, existing benchmarks primarily evaluate hardware description language (HDL) code generation and use waveforms only as supplementary context. This paper presents WaveformQA, an open-source question-answering benchmark for evaluating LLM temporal reasoning over digital waveforms. The benchmark comprises 360 questions with programmatically generated ground truths across eight categories of varying difficulty, including questions targeting multi-signal correlation and event ordering. Waveforms are generated from open-source design implementations, ensuring reproducibility and grounding the benchmark in real hardware behavior. Evaluation of frontier LLMs reveals that while models achieve reasonable accuracy on simple queries, performance degrades due to context window limitations and reasoning difficulties on complex temporal and multi-step questions. In addition, we show that an event-time JSON representation of waveforms improves LLM reasoning accuracy versus the standardized value change dump (VCD) format. The open-source framework supports extending to new question categories and importing new waveform sources, enabling researchers to rapidly prototype temporal reasoning experiments.

时序推理波形分析LLM评测硬件验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。