arXiv:2606.00579cs.CLcs.CV2026-06

仅用文本图像的编码代理也能高效处理音视频任务

Sandboxed Coding Agents are Competitive Omni-modal Task Solvers

论文配图:Sandboxed Coding Agents are Competitive Omni-modal Task Solvers
图 1 · 摘自论文原文
  • 用代码和工具提取音视频片段中的关键信息,转为检索与处理问题
  • 在多个基准上超越主流多模态模型,部分任务表现更优
  • 适合对多模态任务进行低成本、可解释性解决方案的研究者

随着多模态大模型日益聚焦音视频任务,普遍认为需依赖原生多模态模型。我们发现并非如此:仅具备文本+图像输入和沙盒工具接口的编码代理,在多个音视频基准测试中表现媲美甚至超过当前最先进(SOTA)的原生多模态模型及预定义多模态代理框架。轨迹分析表明,其优势源于通过编写代码并调度工具,从字幕、帧及其他模态信号中提取相关证据,将多模态任务转化为检索与信息处理问题,而非直接摄入完整媒体流。我们进一步通过失败分类与过程级追踪分析其局限性,并验证简单技能注入(包括人工编写的和自蒸馏的技能)能显著提升性能。为推动开源技能生成,我们提出 Code-X 训练方案,包含 OmniCoding 轨迹数据集与可验证奖励机制,并在 Qwen-3.5-9B 与 Qwen-3.6-27B 上提供基线。最后,我们主张下一代前沿是多模态处理,引入 TerminalBench-O——一个面向真实世界多模态任务的过程级基准。代码将开源于 https://github.com/Dongping-Chen/OmniCoding。

原文摘要 · Abstract (English)

As multimodal LLMs increasingly target video and audio, it is often assumed that such tasks require native omnimodal models. We show that this is not always the case: coding agents with only text+image access and a sandboxed tool-use interface can match, and in several settings outperform, SOTA native omnimodal models and predefined multimodal agent scaffolds across multiple audio-video benchmarks. Our trajectory analysis suggests that their strength comes from writing code and orchestrating tools to extract relevant evidence from transcripts, frames, and other modality signals, thereby converting omnimodal tasks into retrieval and information-processing problems rather than ingesting entire media streams. We further characterize their limitations through a failure taxonomy and process-level trace analysis, and show that simple skill injection, including human-written and self-distilled skills, substantially improves performance. To explore open-source elicitation, we introduce Code-X, a training recipe with the OmniCoding trajectory dataset and verifiable reward, and provide baselines on Qwen-3.5-9B and Qwen-3.6-27B. Finally, we argue that the next frontier is many-modality processing, and introduce TerminalBench-O, a process-level benchmark for real-world omnimodal processing tasks. Code will be available at https://github.com/Dongping-Chen/OmniCoding.

编码代理多模态音视频工具使用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。