用大模型自动比对全球AI安全政策，发现模型选择影响结果
Automated Analysis of Global AI Safety Initiatives: A Taxonomy-Driven LLM Approach
- 基于统一活动分类体系，用大模型自动提取并映射政策内容
- 五种模型对十份文件比对，相似度得分差异显著，部分配对分歧大
- 适合政策研究者做跨文档对比，但需警惕模型判断偏差
我们提出一种自动化交叉比对框架，用于在共享的活动分类体系下比较两份AI安全政策文件。以《AI安全活动图谱》中的活动类别为固定维度，系统提取并映射相关内容，为每个维度生成每份文件的摘要、简要对比和相似度评分。我们评估了大模型在公共政策文档上的交叉比对稳定性与有效性。使用五种大型语言模型对十份公开文档进行比对，并通过热力图可视化平均相似度得分。结果显示，模型选择显著影响比对结果，部分文档对在不同模型间存在高分歧。三位专家对两组文档进行人工评估，表现出高一致性，但模型评分仍与人类判断存在差异。这些发现支持对政策文件进行多视角对比分析。
原文摘要 · Abstract (English)
We present an automated crosswalk framework that compares an AI safety policy document pair under a shared taxonomy of activities. Using the activity categories defined in Activity Map on AI Safety as fixed aspects, the system extracts and maps relevant activities, then produces for each aspect a short summary for each document, a brief comparison, and a similarity score. We assess the stability and validity of LLM-based crosswalk analysis across public policy documents. Using five large language models, we perform crosswalks on ten publicly available documents and visualize mean similarity scores with a heatmap. The results show that model choice substantially affects the crosswalk outcomes, and that some document pairs yield high disagreements across models. A human evaluation by three experts on two document pairs shows high inter-annotator agreement, while model scores still differ from human judgments. These findings support comparative inspection of policy documents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。