检测视频智能体是否真依赖视觉证据，发现其回答常不靠谱。
Counterfactual Sensitivity Is Not Repairability: Auditing Replay Probes for Video Evidence
- 用破坏性重播对比正常重播，测试答案是否依赖检索的图像。
- 破坏图像后答案改变概率比正常重播高29.3个百分点，效果显著。
- 适合评估视频推理模型的真实性，尤其关注视觉证据依赖性。
使用工具的视频智能体在回答前会检索视觉证据，但最终答案未必依赖这些检索内容。自然的黑盒测试是反事实的：破坏检索帧的语义内容，观察答案是否变化，并与相同流程重播的对照组（SHAM）对比。我们提出CARVE，一种黑盒反事实探测方法，比较在匹配的SHAM和DESTROY重播下答案的变化。在冻结的VideoExplorer型智能体上进行三次独立的k=3运行，DESTROY使答案改变的比例比SHAM高出29.3个百分点，表现出大且可复现的聚合效应。问题级别得分较不稳定，将重播预算从k=3增至k=10虽减少平局，却削弱了原有的零阈值路由策略。在k=3时，CARVE选出1,258个LVBench问题中的538个，准确率提升3.26点，优于多数随机匹配子集。该分数与标注的时间覆盖度关联较弱，因此更适合作为路由信号而非直接的视觉定位分类器。代码已开源于https://github.com/KurbanIntelligenceLab/CARVE。
原文摘要 · Abstract (English)
Tool-using video agents retrieve visual evidence before answering, but the final answer is not forced to depend on what was retrieved. The natural black box test is counterfactual: destroy the semantic content of the frames the agent retrieved and check whether the answer changes, against a matched sham that re-executes the identical pipeline on those same frames. We introduce CARVE, a black-box counterfactual probe that compares answer changes under matched SHAM and DESTROY replays. Across three independent k=3 runs on a frozen VideoExplorer-style agent, DESTROY changes the answer 29.3 percentage points more often than SHAM, yielding a large and reproducible aggregate effect. Question-level scores are less stable, and increasing the replay budget from k=3 to k=10 reduces ties but weakens the original zero-threshold routing policy. At k=3, CARVE selects 538 of 1,258 LVBench questions and improves accuracy by 3.26 points, with higher fallback yield than most matched random subsets. The score shows only a weak association with annotated temporal coverage, so CARVE is best understood as a routing signal rather than a direct grounding classifier. Our implementation is available at https://github.com/KurbanIntelligenceLab/CARVE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。