How Time Distillation Empowers Large Language Models for Time‑Series Forecasting (T‑LLM)

Background

Time‑series forecasting is critical for decision‑making in domains such as finance and large‑scale monitoring. Real‑world forecasting often must operate with limited historical data, making generalization a key challenge. While large‑scale pretrained time‑series models have shown strong performance, the inherent temporal constraints of real‑world data make massive pre‑training costly and less scalable, raising the question of whether generic LLMs can acquire forecasting abilities without relying on extensive time‑series pre‑training.

Problem Definition

Existing approaches either align time‑series representations with LLMs or design specialized architectures, but they do not endow the LLM itself with genuine predictive capability. The authors propose a new training paradigm that treats forecasting as a transferable skill, using structured temporal supervision to teach LLMs to predict.

Method

3.1 Preliminary Definition

Given a multivariate series X = {x_1, x_2, ..., x_L} with C channels, the goal is to predict a future horizon T using a model F(·;θ) that maps past observations to future values.

3.2 T‑LLM Framework Overview

T‑LLM consists of a time‑teacher branch (a compact predictor) and an LLM‑student branch. During training the teacher captures essential temporal patterns; the student LLM learns to mimic the teacher’s predictions. After training the teacher is removed, leaving the LLM as the sole predictor.

3.3 Input Block

Input series are first embedded and projected via a multi‑head attention operator (Q, K, V). The embedded series E_1 is fed to both branches.

3.4 Time‑Teacher Branch

The teacher models trend and frequency components. Trend modeling follows DLinear, decomposing E_1 into trend H_{trend} and residual H_{season}, then applying independent linear projections. Frequency modeling uses a TSLANet‑inspired adaptive spectral block: the series is transformed to the frequency domain, a power spectrum is computed, and a learnable frequency mask compresses the representation. The teacher’s output is the sum of trend and frequency streams, optionally passed through a dominant‑spectrum projection (DSP) module that adapts capacity to the prediction horizon.

3.5 LLM Student Branch

The student uses a pretrained GPT‑2 backbone (first six Transformer layers) fine‑tuned with parameter‑efficient adapters (e.g., LoRA). Its output Y_S is combined with the teacher’s intermediate representations via a lightweight MLP decoder.

3.6 Reverse Distillation

The total loss combines three terms: imitation loss ( L_{imit}) encouraging the student to reproduce the teacher’s predictions, guidance loss ( L_{guide}) providing structured temporal cues at selected depths, and student‑side supervision ( L_{stud}) applied directly to the student’s output. Hyper‑parameters λ_1, λ_2, λ_3 balance these components.

Experiments

4.1 Baselines

T‑LLM is compared against LLM‑based forecasters (CALF, TimeLLM, GPT4TS, UniTime), Transformer models (PatchTST, iTransformer, FEDformer), CNN models (TCN, MICN, TimesNet), MLP models (DLinear, TiDE), and short‑term specialists (N‑HiTS, N‑BETS).

4.2 Implementation Details

The student uses a GPT‑2 model with six layers, Adam optimizer (lr = 0.0005), and loss weights λ_1=1.0, λ_2=0.01, λ_3=1.0. Long‑term tasks employ L1 loss on the ETT dataset and smooth L1 elsewhere; short‑term tasks use SMAPE, MASE, and smooth L1 for feature‑level guidance.

4.3 Long‑Term Forecasting

Evaluated on seven real‑world datasets (four ETT subsets, Traffic, Electricity, Weather) with prediction horizons {96, 192, 336, 720}. Metrics: MSE and MAE. T‑LLM achieves the best or second‑best scores on most datasets, outperforming heavier baselines such as UniTime while remaining lightweight.

4.4 Short‑Term Forecasting

On the M4 dataset (single‑variable series, horizons 6–48), T‑LLM consistently ranks first or second across SMAPE, MASE, and OWA, surpassing CALF thanks to the reverse distillation of predictive behavior.

4.5 Few‑Shot / Zero‑Shot Learning

In a few‑shot regime (10 % of training data), T‑LLM attains top performance on most horizons and datasets, demonstrating robust transferability. In zero‑shot transfer across ETT subsets, T‑LLM reduces error by 2.2 % relative to CALF, indicating that distilled temporal supervision enables the LLM to internalize transferable forecasting skills.

4.6 Efficiency Analysis

Compared to other LLM‑based baselines, T‑LLM has the lowest parameter count and FLOPs while maintaining competitive accuracy, thanks to the removal of teacher components at inference.

4.7 Ablation Studies

Removing L_{stud} degrades performance steadily; disabling L_{imit} or L_{guide} causes noticeable drops, confirming that each loss component contributes uniquely. Head‑tail guidance offers a better trade‑off between accuracy and computational cost than full‑layer guidance.

4.8 Case Study: Epidemic Forecasting

On real‑world flu and COVID‑19 datasets, T‑LLM trained on one disease transfers to the other without further fine‑tuning, achieving the lowest prediction error among baselines and demonstrating strong cross‑domain generalization.

Conclusion

T‑LLM shows that time‑distillation can effectively endow generic LLMs with high‑quality time‑series forecasting capabilities across diverse settings, offering a simple and efficient deployment pipeline without the need for massive time‑series pre‑training.