评测大模型生成以太坊交易代码的准确性与安全性
EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation
- 用动态执行验证生成的交易代码是否正确
- 107个任务测试显示模型在多步骤操作上表现差
- 适合区块链安全研究者和智能合约开发者
大型语言模型正被广泛应用于开发场景，但在链上交易中，哪怕微小错误也可能造成用户不可逆损失。现有评估普遍忽略执行准确性和安全性。我们提出EVM-QuestBench，一个基于执行验证的自然语言到EVM兼容链交易脚本生成基准。该基准采用动态评估：从模板池中采样指令，从预定义区间抽取数值参数，验证器对比实际执行结果与预期值。包含107个任务（62个原子任务，45个复合任务），模块化架构支持快速扩展。运行器在分叉的EVM链上执行脚本，采用快照隔离；复合任务引入步骤效率衰减机制。对20个模型的评估显示显著性能差距，拆分得分揭示单步精确性与多步流程完成度之间的持续不对称性。
原文摘要 · Abstract (English)
Large language models are increasingly applied to various development scenarios. However, in on-chain transaction scenarios, even a minor error can cause irreversible loss for users. Existing evaluations often overlook execution accuracy and safety. We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains. The benchmark employs dynamic evaluation: instructions are sampled from template pools, numeric parameters are drawn from predefined intervals, and validators verify outcomes against these instantiated values. EVM-QuestBench contains 107 tasks (62 atomic, 45 composite). Its modular architecture enables rapid task development. The runner executes scripts on a forked EVM chain with snapshot isolation; composite tasks apply step-efficiency decay. We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion. Code: https://anonymous.4open.science/r/bsc_quest_bench-A9CF/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。