用智能体检索增强交通异常理解，提升视频分析准确性
TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding
- 构建智能体系统，联动图文生成与开放词汇追踪工具检索相关证据
- 在AI City 2026挑战赛中三项任务得分排名第二、第五、第十二
- 适合交通监控、自动驾驶等需要复杂事件理解的场景
交通异常理解（TAU）要求模型能够检测、推理并解释交通视频中的异常事件。为此，我们提出TAU-Agent，一个面向交通异常理解的智能体检索增强框架。给定任务查询后，核心检索智能体协调两个视觉感知工具——视频字幕生成工具和开放词汇追踪工具——以检索并筛选与查询相关的证据，包括文本描述、时间区间和目标轨迹。筛选后的证据连同采样视频帧与输入查询一并输入到经过监督微调的视觉语言模型中，完成最终推理与答案生成。我们在AI City Challenge 2026的域内与域外基准上评估了TAU-Agent，其在Track 3、Track 7和Track 8分别取得0.6779、0.3998和67.9275的分数，排名分别为第二、第十二和第五。代码已公开于https://github.com/siri-rouser/TAU-Agent。
原文摘要 · Abstract (English)
Traffic Anomaly Understanding (TAU) requires models and systems to detect, reason about, and explain anomalous events in transportation videos. To address this challenge, we propose TAU-Agent, an agentic retrieval-augmented framework for traffic anomaly understanding. Given a task query, a central retrieval agent orchestrates two visual perception tools, namely a Video Captioning Tool and an Open-Vocabulary Tracking Tool, to retrieve and select query-relevant evidence, including captions, temporal intervals, and object trajectories. The selected evidence, together with sampled video frames and the input query, is provided to a supervised fine-tuned vision-language model for final reasoning and answer generation. We evaluate TAU-Agent on both the in-domain and the out-of-domain benchmarks from the AI City Challenge 2026. TAU-Agent achieves scores of 0.6779 on Track 3, 0.3998 on Track 7, and 67.9275 on Track 8, ranking second, twelfth, and fifth, respectively. Code is available at: https://github.com/siri-rouser/TAU-Agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。