用可组合流水线自动生成并评分检查清单,提升大模型评估与对齐效率。
AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge
- 设计五类生成抽象,支持灵活构建检查清单流水线。
- 多场景验证显示其评分与人工偏好高度一致。
- 适合需要可解释评估的大模型开发与审稿场景。
检查清单已成为可解释、细粒度评估的热门方法,尤其适用于大模型作为裁判(LLM-as-a-Judge)。除评估外,这些结构化标准还可用于模型对齐、强化学习和自我修正。为此,我们提出 AutoChecklist,一个开源库,将基于检查清单的评估统一为可组合的流水线。核心是一个包含五种检查清单生成抽象的分类体系,每种抽象代表一种不同的评价标准生成策略。模块化生成器→精炼器→评分器流水线可连接任意生成器与统一评分器,新配置仅通过提示模板即可注册。该库内置十种流水线,实现已发表方法,并支持 OpenAI、OpenRouter、vLLM 等多个大模型提供商。除了 Python API,还提供 CLI 工具用于开箱即用评估,以及用于交互式探索的 Web 界面。验证实验表明,这些检查清单方法显著符合人类偏好与质量评分;在 ICLR 同行评审回复案例研究中展示了灵活的领域适应能力。AutoChecklist 已公开:https://github.com/ChicagoHAI/AutoChecklist。
原文摘要 · Abstract (English)
Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly with LLM-as-a-Judge. Beyond evaluation, these structured criteria can serve as signals for model alignment, reinforcement learning, and self-correction. To support these use cases, we present AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines. At its core is a taxonomy of five checklist generation abstractions, each encoding a distinct strategy for deriving evaluation criteria. A modular Generator $\rightarrow$ Refiner $\rightarrow$ Scorer pipeline connects any generator with a unified scorer, and new configurations can be registered via prompt templates alone. The library ships with ten built-in pipelines implementing published approaches and supports multiple LLM providers (OpenAI, OpenRouter, vLLM). Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration. Validation experiments confirm that these checklist methods significantly align with human preferences and quality ratings, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation. AutoChecklist is publicly available at https://github.com/ChicagoHAI/AutoChecklist.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。