arXiv:2608.14015cs.CVcs.AI2026-08

用可审计的工具链实现手术视频长时序推理,无需大量标注数据。

MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning

论文配图:MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning
图 1 · 摘自论文原文
  • 分离推理与感知:文本调度器规划检索,冻结视觉代理执行具体操作。
  • 仅需约100个标注样本,通过奖励驱动的启发式蒸馏构建可复用的检索技能。
  • 在神经外科长视频上表现显著优于现有方法,适合真实医疗场景中的复杂问答。

理解数十分钟的手术视频需要长时序上下文推理,通过在跨阶段的时间证据中定位来回答“之前”、“之后”或“跨越阶段”的问题。现有方法存在局限:单次输入的视觉语言模型(VLM)将整个手术压缩至上下文窗口,丢失关键细节;而训练模型关注位置的视频代理则依赖大量数据,且难以迁移到新领域。我们构建了MedClaw,一个分离推理与感知的代理框架:纯文本协调器规划需收集的证据并生成可审计的工具调用序列,冻结的视觉语言子代理在像素层面执行视图、裁剪、帧检查和外部知识检索。进一步提出无梯度、奖励门控的启发式技能蒸馏循环,从低分轨迹中挖掘技能,并仅当验证奖励提升时保留候选技能,从而生成可复用的检索能力,尤其是定向重看。相比微调权重,通过扩展外部技能库进行适应,仅需约100个标注样例,远低于监督或强化学习微调所需。为评估该代理,我们引入MedClawBench,一个去泄露的、由医生标注的基准,包含1,123个问题,基于自建的长时神经外科录像和预留的公开讲座视频测试集。在两个数据集及全部四个评估维度上,该代理始终优于单次输入的VLM和通用视频代理框架,尤其在长时、跨域神经外科视频上优势明显。

原文摘要 · Abstract (English)

Understanding tens-of-minutes surgical videos requires long-horizon temporal reasoning, answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time. Existing approaches handle this poorly: a one-shot vision-language model (VLM) compresses the whole procedure to fit its context window and loses the detail a "before" or "after" question depends on, while video agents that train the model where to look are data-hungry and transfer poorly to out-of-domain surgery. We build an agent harness that separates reasoning from perception and improves by evolving context rather than optimizing weights. A text-only orchestrator plans which evidence to gather and issues an auditable sequence of tool calls, while frozen vision-language sub-agents execute each call over the pixels, viewing, cropping, inspecting frames, and retrieving external knowledge. We further propose a gradient-free, reward-gated Heuristic Skill Distillation loop that mines the agent's own low-scoring traces and keeps a candidate skill only when it raises a validation reward, yielding reusable retrieval skills, notably directed re-look. Growing an external skill library rather than tuning weights, the loop adapts from only about 100 labeled examples, far fewer than supervised or reinforcement fine-tuning requires. To evaluate this agent, we introduce MedClawBench, a de-leaked, doctor-grounded benchmark of 1,123 questions over self-built long neurosurgery recordings and a held-out public lecture-video test split. Across both datasets and all four evaluation dimensions, our agent consistently outperforms one-shot VLMs and general video-agent frameworks, with the largest gains on the long, out-of-domain neurosurgery videos. Project page: https://fyycs.github.io/medclaw/.

手术视频长时序推理代理系统医疗AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。