构建多模态内容社会价值评估框架,支持量化打分与模型优化。
MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators

- 基于人权公约设计6大维度72项指标,实现多层级价值评估。
- 20亿参数评估器性能接近80亿参数模型,媲美闭源前沿水平。
- 适用于内容安全、伦理审查及模型对齐研究者使用。
评估多模态内容是否契合和平、正义、自由等宏观社会价值观已成为紧迫挑战。现有框架多局限于安全类分类、纯文本心理测量或单标签判断。为此,我们提出MAVEN——一个基于国际人权文件与文化价值理论的层次化评估框架,将价值观划分为6个主维度和72个二级指标,支持多层级定量评分。基于MAVEN,我们构建了人工验证的多模态基准数据集与软匹配评价指标,用于评估视觉语言模型(VLM)在各价值维度上的表现。针对评估器优化,我们提出一种自适应跨度的多级偏好优化方法(SA-MDPO),并设计无需训练的多角色共识推理策略。我们在新基准上评估了多个开源与闭源VLM,揭示其在社会价值判断上的共性与差异。实验表明,我们的紧凑型2B评估器在同系列中达到8B评估器性能,并逼近前沿闭源VLM,为可扩展的社会价值评估提供了实用路径。代码与数据集已开源:https://github.com/zzzzzzzzjj/MAVEN。
原文摘要 · Abstract (English)
Assessing whether multimodal content aligns with macro-societal values, such as peace, justice, and freedom, has become an increasingly urgent challenge. Existing frameworks are largely confined to safety-oriented taxonomies, text-only psychometric probes, or single-label classification. Therefore, we propose MAVEN, a hierarchical framework for macro-societal value evaluation of multimodal content, grounded in international human-rights instruments and cultural value theory. MAVEN organizes values into 6 primary dimensions and 72 secondary indicators, supporting multi-level quantitative scoring. Building on MAVEN, we construct a human-verified multimodal benchmark and a soft-match metric to evaluate VLMs' assessments across value dimensions. For evaluator optimization, we propose a span-adaptive variant of multi-level preference optimization for evaluator distillation, together with a training-free multi-role consensus strategy at inference time. We evaluate existing open- and closed-source VLMs on our benchmark, revealing shared tendencies and clear differences in macro-societal value judgments. Experiments show that our compact 2B evaluator matches its 8B counterpart in the same family and approaches frontier closed-source VLMs, offering a practical path toward scalable macro-societal value evaluation. Our SA-MDPO implementation and MacroValue-Bench are available at https://github.com/zzzzzzzzjj/MAVEN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。