arXiv:2608.20389cs.AI2026-08中稿 · AgentSearch '26 wo…

研究提示:提示中技能暴露方式会影响大模型任务选择,不当设计反致误判。

Representation Affects Retrieval: A Case Study of Skill Discovery and Routing in a Multimodal Agent Harness

  • 用两种形式在提示中展示技能:完整指令(自动加载)与简短列表(按需调用)
  • 全量自动加载时所有任务选对,但默认模式因关键词冲突导致一次错误路由
  • 揭示小规模提示中技能暴露的潜在干扰,对大规模检索系统有启发

一个生产级多模态代理系统需从不断增长的技能库中发现并排序最适配用户任务的技能。小规模下,这一选择通过上下文完成:大模型规划器从系统提示中直接挑选技能表示,不依赖显式的嵌入检索。本文将此上下文选择视为大规模嵌入检索的微观对应,以生产级多模态视频代理系统Tinycloud为案例,分析其技能表示方式。系统将技能分为两类:封装单一外部API或工具的工具型技能(作为基础词汇),以及协调工具调用并生成指定产出的工作流型技能。这些技能通过提示中的两个界面暴露:完整内容的内联体(用于自动加载技能)和单行列表(用于按需调用)。在六个任务上的消融实验显示,全量自动加载能正确选择目标技能;完全关闭自动加载则减慢执行并引发硬性失败;而生产默认设置因关键词冲突,使规划器注意力被自动加载的工具型技能吸引,导致一次工作流技能误选。核心发现是:提示中的技能暴露并非越全越好,部分暴露可能引发词汇竞争,抑制正确选择。该观察与大规模检索式技能路由研究相关,本文定位为案例研究而非基准测试。

原文摘要 · Abstract (English)

A production agent harness must discover and rank, from a growing library of skills, the one most appropriate for a user's task. At small scale this selection happens in context: the LLM planner chooses among skill representations exposed in its system prompt, without an explicit embedding-based retrieval step. We treat this in-context selection as the small-N counterpart to embedding-based skill retrieval at scale, and present a case study of how Tinycloud, a production multimodal video agent harness, represents its skills for the planner. The harness ships skills under two recurring representations: tool-skills that wrap a single external API or system tool and serve as primitive vocabulary, and workflow-skills that orchestrate tool-skill calls plus a template render to produce one named deliverable. The harness exposes them via two surfaces in the system prompt: an inlined-body surface (full instructions, scripts, templates) for autoloaded skills, and a one-line listing for on-demand skills. A six-task selection ablation across three exposure regimes (all-on, default, all-off) shows that full autoload selects the gold skill on every task; all-off slows execution and produces hard discovery failures; and the production default misroutes one task because its lexical signal collides with an autoloaded tool-skill that pulls planner attention away from a listed workflow-skill. The headline finding is that in-prompt exposure of skills is not monotonically helpful: partial exposure can create lexical competition that suppresses correct selection. We connect this small-N observation to recent retrieval-based skill-routing work at large scale, and frame this contribution as a case study rather than a benchmark.

智能体系统技能发现提示工程多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。