对比三种树遍历方法，发现顺序影响模型性能。
Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammars
- 提出新型混合遍历法PRT，结合广度优先与深度优先优势。
- 实验显示不同遍历策略在语法泛化上表现差异显著。
- 适合关注句法结构建模的自然语言生成研究者。
Transformer Grammars（TGs）通过引入句法树结构提升语言建模能力。尽管句法树的线性化方式可能显著影响模型性能，但现有研究仅依赖深度优先遍历（DFT）进行线性化。本文拓展了遍历设计空间，探索广度优先遍历（BFT）及一种新型混合遍历策略——生成规则遍历（PRT），该策略融合了BFT的全局结构前瞻与DFT的早期词元生成优势。我们将这些遍历方法与不同的树结构配置和掩码策略结合，在语言建模、句法泛化和摘要任务上进行实证评估，揭示了嵌套组合与全局前瞻之间的内在权衡，为设计任务感知的Transformer Grammars提供了可操作建议。
原文摘要 · Abstract (English)
Transformer Grammars (TGs) enhance language modeling by incorporating syntactic tree structures. Despite the potentially significant impact on model performance of how syntactic trees are linearized in TGs, existing studies rely solely on Depth-First Traversal (DFT) for linearization. In this paper, we expand the traversal design space by exploring Breadth-First Traversal (BFT) and a novel hybrid traversal strategy, Production-Rule Traversal (PRT), which combines the structural lookahead of BFT with the early lexical generation of DFT. We integrate these traversal methods with varying tree configurations and masking strategies, and empirically evaluate their performance on language modeling, syntactic generalization and summarization. We reveal the inherent trade-offs between nested composition and global lookahead, providing actionable recommendations for designing task-aware Transformer Grammars.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。