测试大模型分析原始电磁信号的能力，发现其在复杂设计任务中表现不佳。
EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals
- 构建多层级基准EMRB，要求模型从原始信号中自主提取特征并编程求解。
- 14个模型平均得分仅21.2%，复杂系统设计题正确率远低于基础测量题。
- 提出ReconPilot方法，通过分步推理提升模型性能，适合信号处理与AI结合研究者。
大型语言模型（LLMs）被广泛用于科学与工程代码代理，但其对原始物理层测量数据的分析能力尚未验证。本文提出电磁推理基准EMRB，评估模型通过编写和运行代码分析原始I/Q数据的能力。该基准包含200道题，覆盖五个难度等级和27种题型，涵盖信号检测到OFDM设计等任务，基于11种真实信号类型生成且具有已验证真值。不同于依赖预处理特征或结构化表格的基准，EMRB仅提供原始采集数据，问题涉及的量需通过代码自行发现。我们评估了14个主流模型（包括商用、开源及推理优化型），得分介于24.1%至78.9%之间，平均得分从基础测量的84.9%降至系统设计的21.2%。此外，我们提出结构化方法ReconPilot，分离信号侦察、定向分析与自验证步骤，在三个模型骨干上使总分提升3.8至17.6分，并改善13/15种组合。所有数据与代码已公开于GitHub。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used as code agents for scientific and engineering analysis, but their ability to analyze raw physical-layer measurements remains untested. We introduce \textbf{EMRB} (\textbf{E}lectro\textbf{m}agnetic \textbf{R}easoning \textbf{B}enchmark), which evaluates whether LLMs can analyze raw I/Q data by writing and running code. EMRB contains 200 problems across five difficulty levels and 27 question types, from signal detection to OFDM design, generated from 11 signal types with verified ground truth. Unlike benchmarks built on preprocessed features or structured tables, EMRB provides only the raw capture; the quantities each question refers to must first be discovered through code. We evaluate 14 LLMs spanning proprietary, open-weight, and reasoning-oriented families. Scores range from 24.1\% to 78.9\%, with the mean dropping from 84.9\% on basic measurement to 21.2\% on system design. We also propose \textbf{ReconPilot}, a structured method that separates signal reconnaissance, targeted analysis, and self-verification. Across three backbones, ReconPilot raises the overall score by 3.8 to 17.6 points and improves 13 of 15 backbone-level combinations tested. All data and code are publicly released in \href{https://github.com/mingxuZhang2/EMRB}{\textcolor{blue}{our GitHub repository}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。