提出一种类小波多尺度语言理解架构,提升效率与长文本建模能力。
Hierarchical Resolution Transformers: A Wavelet-Inspired Architecture for Multi-Scale Language Understanding
- 通过多分辨率注意力机制,从字符到语篇分层处理语言
- 在多个基准上平均性能超越Transformer 3.8%~6.1%,内存减少42%
- 适合需要高效处理长文本的场景,如文档理解、对话建模
Transformer架构在自然语言任务中表现优异,但将文本视为扁平的词元序列,忽略了人类语言的层次结构,导致计算复杂度为二次方,组合泛化能力弱,且缺乏语篇级建模。本文提出层次分辨率变压器(HRT),一种受小波启发的神经架构,可同时在多个分辨率下处理语言,涵盖从字符到语篇单位。HRT构建多分辨率注意力,支持自底向上组合与自顶向下上下文感知。通过指数级序列压缩,实现O(nlogn)复杂度,显著优于标准Transformer。在GLUE、SuperGLUE、Long Range Arena和WikiText-103等基准上的实验表明,HRT在相同参数量下,相比BERT和GPT类模型,平均性能提升3.8%~6.1%,内存降低42%,推理延迟减少37%。消融实验验证了跨分辨率注意力与尺度特异性模块的独立贡献。结果证明,HRT是首个在计算结构上契合人类语言层次组织的架构,证实多尺度、类小波处理可带来理论效率与实际性能的双重提升。
原文摘要 · Abstract (English)
Transformer architectures have achieved state-of-the-art performance across natural language tasks, yet they fundamentally misrepresent the hierarchical nature of human language by processing text as flat token sequences. This results in quadratic computational cost, weak computational cost, weak compositional generalization, and inadequate discourse-level modeling. We propose Hierarchical Resolution Transformer (HRT), a novel wavelet-inspired neural architecture that processes language simultaneously across multiple resolutions, from characters to discourse-level units. HRT constructs a multi-resolution attention, enabling bottom-up composition and top-down contextualization. By employing exponential sequence reduction across scales, HRT achieves O(nlogn) complexity, offering significant efficiency improvements over standard transformers. We evaluated HRT on a diverse suite of benchmarks, including GLUE, SuperGLUE, Long Range Arena, and WikiText-103, and results demonstrated that HRT outperforms standard transformer baselines by an average of +3.8% on GLUE, +4.5% on SuperGLUE, and +6.1% on Long Range Arena, while reducing memory usage by 42% and inference latency by 37% compared to BERT and GPT style models of similar parameter count. Ablation studies confirm the effectiveness of cross-resolution attention and scale-specialized modules, showing that each contributes independently to both efficiency and accuracy. Our findings establish HRT as the first architecture to align computational structure with the hierarchical organization of human language, demonstrating that multi-scale, wavelet-inspired processing yields both theoretical efficiency gains and practical improvements in language understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。