自动生成针对特定任务的评估工具，解决大模型应用中的幻觉问题
TaskEval: Synthesised Evaluation for Foundation-Model Tasks
- 用通用元模型捕捉任意大模型任务特征
- 通过高效人机交互机制生成高精度评估方案
- 适合需要定制评估流程的工程团队使用
大模型应用中的幻觉问题是核心挑战。理解这些细微故障的发生位置和方式依赖于称为“evals”的评估方法。以往研究集中于为特定任务定义新评估方法或基准数据集，但当缺乏相应指标或数据集时，软件团队仍难以评估任务。本研究提出一种合成特定任务评估程序的方法，实现自动化与定制化用户界面结合，支持人工反馈采集。核心创新包括：（1）可泛化到任意任务的元模型；（2）高效的人工反馈交互协议；（3）自动选择或生成合适评估项的评估合成器。我们在ToolName中实现该方法，并在图表数据提取与文档问答两个不同任务上验证概念。初步评估显示所选评估项准确率分别达到93%和90%。本研究回应了工程团队面临的日益严峻的问题：如何有效评估和审查大模型输出。
原文摘要 · Abstract (English)
Hallucinations are a key concern when creating applications that rely on Foundation models (FMs). Understanding where and how these subtle failures occur in an application relies on evaluation methods known as \textit{evals}. Prior work focuses on defining new eval methods or benchmark datasets for specific tasks. However, neither helps a software team with a task-specific FM application when there is no metric or dataset. The demand for both automated approaches and deep integration of human insight makes this a challenging problem. We address this gap by proposing an approach to synthesise a FM task-specific evaluator program that provides automation and a custom UI for capturing feedback. The core novelty of our approach lies in: (1) a task-agnostic meta-model that captures properties of any FM task, (2) an interaction protocol for efficient use of human feedback, and (3) an eval synthesiser that selects or generates an appropriate set of evals. We implement our approach in \toolname and demonstrate the concept on two diverse FM tasks: chart data extraction and document question answering. A preliminary evaluation on the quality of our selected evals shows 93\% and 90\% accuracy respectively. Our research tackles a growing problem facing engineering teams, how to evaluate and review outputs from FM tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。