用大模型自动分析模型差异，让复杂对比变简单。
VERBA: Verbalizing Model Differences Using Large Language Models
- 用大模型采样并生成模型间差异的自然语言描述。
- 对性能差5%但行为差20-25%的决策树模型，准确率达80%。
- 适合需要快速理解模型差异的研究者和开发者。
当前机器学习中存在‘模型泛滥’现象：同一任务下众多模型表现相近但行为不同。用户在选型时依赖模型对比文档，但每对模型都需比较，数量达O(N²)，人工难以处理。为此，我们提出VERBA，利用大语言模型（LLM）通过采样两模型输出，生成其差异的自然语言描述。我们建立评估协议，通过仿真验证描述的有用性，并构建包含多种常用模型的基准测试集。对于性能差异≤5%但行为差异达20-25%的决策树模型对，VERBA生成的差异描述准确率可达80%；若引入模型结构信息，准确率进一步提升至90%。VERBA为事后提升模型透明性与可比性开辟了新路径。
原文摘要 · Abstract (English)
In the current machine learning landscape, we face a "model lake" phenomenon: Given a task, there is a proliferation of trained models with similar performances despite different behavior. For model users attempting to navigate and select from the models, documentation comparing model pairs is helpful. However, for every $N$ models there could be $O(N^2)$ pairwise comparisons, a number prohibitive for the model developers to manually perform pairwise comparisons and prepare documentations. To facilitate fine-grained pairwise comparisons among models, we introduced $\textbf{VERBA}$. Our approach leverages a large language model (LLM) to generate verbalizations of model differences by sampling from the two models. We established a protocol that evaluates the informativeness of the verbalizations via simulation. We also assembled a suite with a diverse set of commonly used machine learning models as a benchmark. For a pair of decision tree models with up to 5% performance difference but 20-25% behavioral differences, $\textbf{VERBA}$ effectively verbalizes their variations with up to 80% overall accuracy. When we included the models' structural information, the verbalization's accuracy further improved to 90%. $\textbf{VERBA}$ opens up new research avenues for improving the transparency and comparability of machine learning models in a post-hoc manner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。