构建真实以太坊交易意图的评测基准，检验大模型将自然语言转为正确链上操作的能力。
Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions
- 基于真实以太坊主网数据构建29,921条单步、1,575条多步指令
- 发现多数模型生成语法正确但无法达成预期状态变更的交易
- 适合研究Web3智能代理、大模型执行推理能力的开发者
大型语言模型（LLMs）为Web3提供了变革性接口，但现有评测难以捕捉将高层次用户意图转化为功能正确、依赖状态的链上交易的复杂性。本文提出 extsc{Intent2Tx}，一个高保真度基准，包含从300天真实以太坊主网交易中提取的29,921个单步和1,575个多步实例。不同于依赖合成指令的先前工作， extsc{Intent2Tx}基于11类真实协议交互，涵盖多样化的长尾去中心化金融（DeFi）原语。为实现严谨评估，我们设计了执行感知框架，通过分叉主网环境的差分状态分析，超越表面文本匹配。对16个前沿大模型的广泛测试表明，规模扩展和检索增强可提升逻辑一致性和参数精度，但当前模型在分布外泛化和多步规划上仍存短板。关键发现：语法正确的输出常无法实现预期状态转移，凸显当前‘推理到执行’能力的重大差距。该基准为构建以意图为中心的可靠自治代理奠定基础。代码与数据：https://anonymous.4open.science/r/Intent2Tx_Bench-97FF。
原文摘要 · Abstract (English)
The emergence of Large Language Models (LLMs) offers a transformative interface for Web3, yet existing benchmarks fail to capture the complexity of translating high-level user intents into functionally correct, state-dependent on-chain transactions. We present \textsc{Intent2Tx}, a high-fidelity benchmark featuring 29,921 single-step and 1,575 multi-step instances meticulously derived from 300 days of real-world Ethereum mainnet traces. Unlike prior works that rely on synthetic instructions, \textsc{Intent2Tx} grounds natural language intents in real-world protocol interactions across 11 categories, including diverse long-tail Decentralized Finance (DeFi) primitives. To enable rigorous evaluation, we propose an execution-aware framework that transcends surface-level text matching by employing differential state analysis on forked mainnet environments. Our extensive evaluation of 16 state-of-the-art LLMs reveals that while scaling and retrieval-augmentation enhance logical consistency and parameter precision, current models struggle with out-of-distribution generalization and multi-step planning. Crucially, our execution-based analysis demonstrates that syntactically valid outputs often fail to achieve intended state transitions, highlighting a significant gap in current "reasoning-to-execution" capabilities. \textsc{Intent2Tx} serves as a critical foundation for developing autonomous, reliable agents in intent-centric Web3 ecosystems. Code and data: https://anonymous.4open.science/r/Intent2Tx_Bench-97FF .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。