用AI辅助标注工具提升信息检索数据集构建效率
AIANO: Enhancing Information Retrieval with AI-Augmented Annotation
- 结合人类判断与大模型建议，实现智能协同标注
- 标注速度提升近一倍，准确率与易用性同步提高
- 适合需要高质量检索数据的科研与工程团队
大型语言模型（LLM）和检索增强生成（RAG）的兴起，极大提升了对高质量、精心标注的信息检索数据集的需求。然而，当前的数据集构建仍依赖通用标注工具，导致标注过程复杂且低效。为此，我们开发了专用标注工具AIANO，采用紧密融合人类专家与大模型建议的AI增强型标注流程，使标注者在保留决策控制权的同时利用AI建议。在一项包含15名参与者的自身对照实验中，使用基准工具与AIANO分别构建问答数据集。结果显示，AIANO将标注速度几乎提升一倍，同时更易使用并提升了检索准确率。这些结果表明，AIANO的AI增强方法显著加速并优化了信息检索任务的数据集创建，推动了高依赖检索领域中的标注能力发展。
原文摘要 · Abstract (English)
The rise of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) has rapidly increased the need for high-quality, curated information retrieval datasets. These datasets, however, are currently created with off-the-shelf annotation tools that make the annotation process complex and inefficient. To streamline this process, we developed a specialized annotation tool - AIANO. By adopting an AI-augmented annotation workflow that tightly integrates human expertise with LLM assistance, AIANO enables annotators to leverage AI suggestions while retaining full control over annotation decisions. In a within-subject user study ($n = 15$), participants created question-answering datasets using both a baseline tool and AIANO. AIANO nearly doubled annotation speed compared to the baseline while being easier to use and improving retrieval accuracy. These results demonstrate that AIANO's AI-augmented approach accelerates and enhances dataset creation for information retrieval tasks, advancing annotation capabilities in retrieval-intensive domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。