构建动态实时问答评测框架,检验模型获取最新信息能力
Benchmarking Real-Time Question Answering via Executable Code Workflows

- 用可执行代码自动爬取网页并提取答案,生成实时真值
- 顶尖模型实时问答准确率仅46%,暴露严重时间适应缺陷
- 适合研究智能体实时推理与时间状态管理的学者
检索实时信息是搜索集成智能体在现实应用中的核心能力。然而,现有基准大多为静态,无法捕捉信息的时间动态性与知识的持续演变。为此,我们提出RT-QA,一个基于可执行代码工作流的动态评估框架,可在评估时实时获取最新答案。具体地,构建了由智能体驱动的流水线,自主生成代码进行网络爬取与基于DOM的答案提取,以生成实时真值。为确保长期评估稳健性,该流水线还引入自修复机制,以适应网页结构变化。RT-QA涵盖12个领域(如金融、体育),包含320个中文问题,按难度分为三类。对前沿模型(如GPT-5.2、GLM-4.7)的广泛评估显示其在实时适应性上存在显著局限:最优模型准确率仅为46%。分析揭示两大主要失败模式:(1) 懒惰检索——智能体依赖搜索摘要而非深入特定网站查找信息(占失败的20%);(2) 时间混淆——智能体错误使用历史日期(如2024年事件)而未重新锚定至当前时间(2026年)进行后续推理。这些发现表明,未来智能体不仅需更优检索策略,还需具备稳健的时间状态管理能力。
原文摘要 · Abstract (English)
Retrieving real-time information is a fundamental capability for search-integrated agents in real-world applications. However, existing benchmarks are predominantly static and therefore fail to capture the temporal dynamics of information and the continuously evolving nature of real-world knowledge. To address this limitation, we propose RT-QA, a dynamic evaluation framework that leverages executable code workflows to retrieve up-to-date answers at evaluation time. Specifically, we construct an agent-driven pipeline that autonomously generates code for web crawling and DOM-based answer extraction to produce real-time ground truth. To ensure robust evaluation over time, the pipeline further incorporates a self-repair mechanism to adapt to changes in web page structures. RT-QA spans 12 domains (e.g., Finance, Sports) with 320 Chinese questions categorized into three difficulty levels. Extensive evaluations of state-of-the-art models (e.g., GPT-5.2, GLM-4.7) reveal significant limitations in real-time adaptability: even the best models achieve only 46% accuracy. Our analysis highlights two primary failure modes: (1) Lazy Retrieval, where agents rely on search snippets instead of deeply scanning specific websites for information (20% of failures); and (2) Temporal Confusion, a cognitive error where agents retrieve a historical date (e.g., an event in 2024) and fail to re-anchor to the current time (2026) for subsequent reasoning. These findings suggest that future agents require not just better retrieval strategies, but robust temporal state management.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。