让用户通过交互式反思,逐步明确模糊的视觉概念定义。
Agile Deliberation: Concept Deliberation for Subjective Visual Classification
- 分两阶段迭代优化:先拆解概念,再用边缘案例引导用户修正理解。
- 相比自动分解和手动讨论,F1得分提升超7.5%且认知负担更低。
- 适合内容审核、创意筛选等需主观判断的视觉分类场景。
从内容审核到内容筛选,需要对视觉概念进行分类的应用正在快速增长。现有的人机协同方法通常假设用户已有清晰稳定的概念理解,能提供高质量标注。但现实中,用户常从模糊想法出发,需通过‘概念思辨’不断细化。我们通过对内容审核专家的结构化访谈发现这一实践,并将其策略转化为名为“敏捷思辨”的人机协同框架,专门支持动态演变和主观性概念的构建。系统通过两个阶段实现:(1) 概念界定,将初始概念分解为子概念层级结构;(2) 概念迭代,展示语义边界案例,引导用户反思与反馈,持续对齐图像分类器与用户意图。由于概念思辨本质主观且互动性强,我们进行了18次用户会话实验,每次1.5小时,而非依赖标准数据集评估。结果表明,该框架相较自动分解基线提高7.5% F1得分,比人工思辨高出3%以上,同时用户报告概念理解更清晰、认知负荷更低。
原文摘要 · Abstract (English)
From content moderation to content curation, applications requiring vision classifiers for visual concepts are rapidly expanding. Existing human-in-the-loop approaches typically assume users begin with a clear, stable concept understanding to be able to provide high-quality supervision. In reality, users often start with a vague idea and must iteratively refine it through "concept deliberation", a practice we uncovered through structured interviews with content moderation experts. We operationalize the common strategies in deliberation used by real content moderators into a human-in-the-loop framework called "Agile Deliberation" that explicitly supports evolving and subjective concepts. The system supports users in defining the concept for themselves by exposing them to borderline cases. The system does this with two deliberation stages: (1) concept scoping, which decomposes the initial concept into a structured hierarchy of sub-concepts, and (2) concept iteration, which surfaces semantically borderline examples for user reflection and feedback to iteratively align an image classifier with the user's evolving intent. Since concept deliberation is inherently subjective and interactive, we painstakingly evaluate the framework through 18 user sessions, each 1.5h long, rather than standard benchmarking datasets. We find that Agile Deliberation achieves 7.5% higher F1 scores than automated decomposition baselines and more than 3% higher than manual deliberation, while participants reported clearer conceptual understanding and lower cognitive effort.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。