TimeHAT: A Hybrid-Attention Pretraining Framework with Point-Patch Masking for Time Series
ORIGINAL / Learning from Points and Patches: A Pre-trained Hybrid Attention Transformer for Time Series Analysis
The paper proposes TimeHAT, a general-purpose time-series pretraining framework whose unified point-patch masking supervision covers both fine-grained fluctuations and broader temporal patterns, and whose hybrid attention encoder aims to mitigate rank degeneration; it is evaluated on classification, long-term forecasting, imputation, and anomaly detection.
01 ABSTRACT
The authors propose TimeHAT, which unifies point-level and patch-level masking into a single pretraining objective combining contextual reconstruction and causal extrapolation. The method stacks prefix-constrained bidirectional attention in early layers with directional hybrid attention in deeper layers to retain contextual interaction while mitigating rank degeneration. Experiments span four downstream tasks under fine-tuning, few-shot, and zero-shot settings, with competitive performance reported as the authors' claim.
02 KEY FINDINGS
- The Unified Point-Patch Masking Strategy (UPPMS) provides both point-level and patch-level supervision, capturing fine-grained variation and broader temporal patterns.
- The Hybrid Attention Encoder (HAE) uses prefix-constrained bidirectional attention in initial layers and directional hybrid attention in deeper layers to mitigate rank degeneration.
- Evaluation covers classification, long-term forecasting, imputation, and anomaly detection, including fine-tuning, few-shot, and zero-shot transfer settings.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.LG