TiRex: LSTMs Take The Lead Again in Time Series Forecasting
A foundation model that challenges the status-quo in time series
Let’s rewind a decade to the early days of Deep Learning for time series.
When Keras launched, the internet was filled with sketchy tutorials using LSTMs for forecasting. Some were so bad they applied LSTMs to financial data with one-step-ahead predictions. The result was nice-looking plots and extra blog traffic, but totally useless in practice.
LSTMs’ reputation suffered. Yet, when used correctly, they can be highly effective—handling multivariate, multi-step outputs without the preprocessing that statistical models and tree-based methods need. Models like TFT even used them in hybrid architectures for extra performance
Today, the strongest deep learning forecasters are usually MLP-based linear models, and to a lesser extent, Convnets and Transformers.
Professor Sepp Hochreiter, one of the co-authors of LSTMs, leads a research group that is now a promising startup aiming to revisit and upgrade LSTMs. One focus is time series forecasting in the pretrained category, with their new model TiRex [1].
This article explains:
How TiRex works,
How xLSTMs improve on traditional LSTMs,
What sets TiRex apart from other time-series foundation models,
Why TiRex excels with sparse data and missing values.
✅ For a hands-on project to using TiRex on sparse data, check the AI Projects Folder (Project 21) and the companion article here.
Preliminaries - LSTM
The classic LSTM contains 2 main signals:
Long-term memory: The cell state c, which carries information across multiple time steps.
Short-term memory: The hidden state h, which through the input and forget gates, decides what will be overwritten in c, and through the output gate decides what passes to the next cell.
The input, output, and forget gates are shown more clearly in Figure 2:
LSTMs, first published in 1997, became a key model in deep learning and rose to prominence around 2015.
LSTMs have 2 major pros and cons:
Pros:
Efficient inference: They process sequences step by step, keeping a hidden state indefinitely. This enables efficient long-sequence processing with near-constant memory.
Unbounded generation: They can generate outputs indefinitely by updating the hidden state regardless of input length.
Cons:
Limited context: The hidden state updates at each step, giving only indirect access to past words.
Training bottlenecks: No parallelization in training due to sequential updates, leading to vanishing or exploding gradients.
The limited context is the biggest issue. LSTMs/RNNs process one word at a time and don’t have direct access to previous words, only to the compressed information in the hidden state.
This is especially challenging for NLP tasks because the LSTM must determine in advance which information to retain in the hidden state for future use. That’s why LSTMs are a more natural fit for time series, where data is less dense.
Check the Appendix of this article for a comparison of LSTMs vs Transformers.
Preliminaries - xLSTM
The authors extend the original LSTM with a new design called xLSTM (extended LSTM).
This architecture consists of 2 modules: sLSTM (scalar LSTM) and mLSTM (memory LSTM). See Figure 3:
Although TiRex uses only sLSTM (shows better performance on benchmarks - more below), let’s briefly cover both.
The sLSTM also uses a different activation, the Exponential Gating. The issue with sigmoid is that it squashes values into [0,1], losing information as gradients pass forward.
Exponential Gating (exp) removes this limit, but activations can become large. To control this, sLSTM adds a normalizer state (nt) that stabilizes the intermediate hidden state outputs:
The mLSTM also uses Exponential Gating but expands the memory cell from a scalar c to a matrix C. This matrix stores vectors k and v (keys and values, like in Transformers) and q (queries) for retrieval.
Retrieval is performed using the Bidirectional Associative Memories concept (BAM). The corresponding hidden state is calculated (retrieved) by multiplying the state matrix C by the corresponding query.
Most importantly, mLSTM is fully parallelizable. Notice mLSTM’s state equations below - there is no ht-1 dependence anywhere, hence the current step doesn’t have to wait for the hidden states of the previous step.
Feel free to check the paper for more details. Write in the comments if you’d like a deeper dive into xLSTM.
Finally, xLSTM also leverages standard deep learning features, such as Layer Normalization and Swish activations. TiRex adds extra optimizations for time series, which we’ll analyze below.
What is TiRex?
TiRex is a decoder-only xLSTM-based foundation model that achieves superior performance in time-series forecasting
Key Characteristics:
Pretrained on diverse data: Chronos, GIFT-Eval, and synthetic datasets.
Lightweight: Contains just 35M parameters, far smaller than models like Chronos-Bolt-Base (200M), Toto (151M), and TimesFM-2.0 (500M).
Superior Performance: Currently ranks #1 on the GIFT-Eval benchmark, surpassing Toto within a month. The competition is tight — a sign of rapid progress in this space!
Probabilistic forecasting: Predicts 9 quantiles, enabling decision-aware forecasting.
Figure 6 shows the TiRex architecture. Besides the xLSTM (essentially only an sLSTM), it includes MLPs and residual blocks with skip connections:
TiRex introduced a few innovations that make it unique. Let’s view them:
Multi-Patch Forecasts
Like other foundation models, TiRex partitions timepoints into patches — similar to tokens in language models.
It also handles missing values by creating a binary mask, which is concatenated with the input.
Both input and output patches have a length of 32 (chosen after tuning). What happens if the forecast horizon exceeds 32? Many models, like TimesFM, autoregressively feed back predictions to forecast the next patch.
TiRex takes a different route. It treats future inputs as missing values. Instead of feeding back predictions, it marks future steps as missing (zeros in the mask) and lets the xLSTM hidden state carry forward both the predictive “median” and its uncertainty.
This little tweak makes TiRex efficient and effective with sparse or intermittent data. To my understanding, the patch masking technique brings 2 key benefits:
Uncertainty propagation: The model never commits to a single point estimate at patch boundaries. Its hidden state carries the full predictive distribution forward.
Coherence: The second-patch quantile forecasts are conditioned not on a single leaked estimate, but on the entire posterior carried forward. This yields tighter, more coherent probabilistic forecasts and avoids the “variance collapse” or “re-initialization” we get when we feed back raw medians.
Contiguous Patch Masking
Pretraining in time series foundation models has come a long way. Older models usually masked the last window or random values.
TiRex introduced a new technique called Contiguous Patch Masking (CPM) - Figure 7:
CPM improves stable multi-patch prediction by masking entire consecutive segments of a time series during pre-training. Unlike random token-level masking, CPM hides full patches, treating them as missing values, which closely mirrors the inference setting where multiple future patches must be forecasted.
The process works as follows: for each training sequence, the model samples the number of consecutive patches to mask (c_mask) and the probability of masking (p_mask). A binary mask is then applied across the series, expanded so that masked regions cover contiguous spans of patches. Because adjacent regions may overlap, the actual masked span can exceed c_mask.
While CPM borrows ideas from BERT-style masking, its training objective aligns more with causal, decoder-only models since predictions depend on shifted targets and unidirectional information flow (remember, TiRex behaves like a decoder-only model).
Another model that had used a similar approach was MOMENT. However, MOMENT was an encoder-only model (more natural for masked encoding) and didn’t focus on continuous patches.
The masking patch technique for missing values and CPM makes TiRex especially suitable for industrial data and significantly better at handling sparse sequences than other foundation models. Figure 8 illustrates an example of using TiRex on synthetic data.
Training Data Augmentation
Another area where foundation models focus is on data augmentation techniques, including synthetic data.
Data augmentation is highly beneficial, in fact each new foundation time series model builds on the augmentation strategies of the previous SOTA models.
To diversify time series patterns, TiRex introduces 3 augmentation types during pre-training, shown in Figure 9:
Amplitude Modulation scales values through linear trends with potential change points, creating shifted dynamics.
Censor Augmentation clips values at a random threshold drawn from the series’ empirical distribution.
Spike Injection overlays periodic spikes defined by kernels such as tophat, RBF, or linear, with parameters sampled per instance.
These augmentations are applied probabilistically (50% for amplitude and censoring, and 5% for spike injection), ensuring the model learns from a broad range of temporal variations.
TiRex further augments pretraining data by applying the TSMixup and KernelSynth processes from Chronos. TSMixup creates new time series by randomly selecting base sequences from different datasets and blending them through convex combinations. KernelSynth is for generating diverse synthetic data using Gaussian processes
TiRex is further refined by using instance normalization (which is very beneficial for time series). The model supports a maximum context length of 2048.
Evaluating TiRex
The authors evaluate TiRex on 2 popular benchmarks. Chronos-ZS (Chronos Zero-Shot) and GiftEval-ZS
Chronos-ZS covers 27 short-term forecasting datasets, while GiftEval spans 24 datasets and 97 settings across short, medium, and long-term horizons. TiRex’s training data does not overlap with these, ensuring no data leakage. Thus, overlapping cases in GiftEval were excluded, yielding the GiftEval-ZS benchmark.
Performance is measured with mean absolute scaled error (MASE) for point forecasts and continuous ranked probability score (CRPS), approximated with weighted quantile loss (WQL). Scores are normalized against a seasonal naïve baseline and aggregated for fair comparison.
TiRex is benchmarked against zero-shot models such as Chronos, TimesFM, and MOIRAI, and task-specific models like PatchTST and DeepAR
GIFT-Eval Benchmark
Data leakage is a major issue when evaluating pretrained models. The TiRex paper points out which models suffer from this problem in the benchmarks below:
We observe the following:
TiRex leads the GiftEval-ZS benchmark, clearly ahead of the next best zero-shot models.
It ranks highest overall, with a notable gap above TimesFM-2.0, TabPFN, and Chronos-Bolt-Base, which cluster closely together.
Unlike other models that specialize in short or long-term forecasting, TiRex excels at both.
Despite using just 35M parameters, TiRex outperforms much larger models and becomes the first zero-shot model to beat PatchTST and TFT in long-term tasks.
Statistical models are significantly outperformed
Chronos-ZS benchmark
Like in GiftEval-ZS, TiRex shows consistent performance patterns.
TiRex ranks best in WQL score and overall rank, and second in MASE, just behind TabPFN-TS.
Notice that MOIRAI scores higher on Chronos-ZS, but this is largely due to 82% overlap with its pre-training data (data-leakage issue)
These results confirm TiRex’s robustness in zero-shot generalization.
Some general remarks:
The benchmarks are updated with the latest versions of foundation models (e.g., TimesFM-2.0), making the comparison among foundation models competitive.
Unfortunately, no Tree models (e.g., LightGBM) are included, possibly because the evaluation focuses on univariate forecasting.
In the updated GIFT-Eval benchmark, at the time of writing, TiRex is flagged for TestData Leakage. The authors removed datasets from the evaluation set that overlap with the pretraining set. For example, the Temperature-Rain dataset, included in Chronos-Train, is removed from the GiftEval test.
The data-leakage flag may also relate to the Electricity-Daily dataset in GIFT-Eval-Test, while Electricity-Weekly/Hourly/15-T appear in Chronos-Train. In my view, this does not constitute true leakage, as different frequencies create distinct datasets with different statistical properties.
Fine-Tuning TiRex
The authors have not yet released fine-tuning or pretraining scripts. Their GitHub README mentions that these will be available in the future.
In any case, they do include a fine-tuning example, comparing TiRex with Tiny-Time-Mixers-R2 (TTM-R2) on the GiftEval dataset, using the same metrics as before.
Essentially, both models are further finetuned on the training split of the GiftEval benchmark, with TTM-R2 having a slight advantage due to some minor data overlap.
Eventually, we notice that TiRex outperforms TTM-r2 - but most importantly, finetuning is beneficial for TiRex.
When the fine-tuning scripts become available, I’ll update the notebook in the AI Projects Folder, so stay tuned!
Thank you for Reading!
Horizon AI Forecast is a reader-supported newsletter, featuring occasional bonus posts for supporters. Your support through a paid subscription would be greatly appreciated.
Appendix
Table 1 shows the differences between Transformers and RNNs:
References
[1] Auer, Andreas; Podest, Patrick; Klotz, Daniel; Böck, Sebastian; Klambauer, Günter; Hochreiter, Sepp. TiRex: Zero‑Shot Forecasting Across Long and Short Horizons with Enhanced In‑Context Learning (May 2025).
[2] Guillaume Chevalier, Schematic of the Long-Short Term Memory cell, a component of recurrent neural networks
[3] Kjærran, Adrian; Bugge, Erling Stray; Vennerød, Christian Bakke. Time Series – Long Short-term Memory RNN (2021)
[4] Beck, Maximilian; Pöppel, Korbinian; Spanring, Markus; Auer, Andreas; Prudnikova, Oleksandra; Kopp, Michael; Klambauer, Günter; Brandstetter, Johannes; Hochreiter, Sepp. xLSTM: Extended Long Short‑Term Memory (NeurIPS 2024)