用大模型生成技能包,无法提升数据科学任务表现。
Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows
- 测试大模型自动生成的技能包在四类数据任务中的效果
- 7560次实验显示生成技能无显著提升,最高仅1.2个百分点差异
- 即使拆解技能组件或对比无关格式内容,效果仍无差别
产品数据科学家常依赖基于大模型的智能体处理重复性任务,如数据清洗、编写SQL、选择统计检验方法和结果格式化。为减少重复提示,可使用可复用的技能文件封装任务指导。专家编写的技能虽质量高,但跨众多任务家族维护成本大。本文探讨大模型自动生成的技能是否能作为低人工成本替代方案:相比仅用任务提示,生成技能能否提升性能?我们在数据准备、提取、统计分析和报告四个生命周期阶段分别测试一个生成技能。结果表明,与仅使用任务提示相比,完整生成技能并未带来可靠性能提升。进一步通过消融实验分析技能各组件,涵盖56个任务、9种模型配置和3个提供商,共7560次运行。所有变体与基准对比的p值均不低于0.396,整体差异仅为1.2个百分点。补充的令牌匹配对照实验(1512次运行)发现,完整技能与无关格式内容效果相当。研究警示:不应默认采用单次生成技能作为数据科学工作流的通用提示策略。
原文摘要 · Abstract (English)
Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We ask whether LLM-generated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone? We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. We find no reliable improvement from full generated skills over No-Skill prompting. We then ask whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all p-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。