DeepFusion: A Multi-Modal Deep Fusion Network for Short-Term Heavy Rainfall Forecasting

Short‑term heavy precipitation is a critical weather hazard that threatens urban safety and daily life; accurate 0‑2 hour forecasts are essential for flood control and travel planning.

The task is challenging because (1) multiple heterogeneous data sources—radar, precipitation observations, numerical weather prediction (NWP) fields, and terrain—must be fused effectively, and (2) precipitation systems exhibit strong multi‑scale spatial and non‑linear temporal dependencies.

Model Design

Data Features

Four data modalities are integrated:

Radar data : six 6‑minute radar echo frames (past 36 minutes) capture the instantaneous state of precipitation systems.

Historical precipitation : the same 36‑minute window of precipitation intensity provides direct temporal evolution signals.

NWP data : eleven physically meaningful variables (e.g., Q1000, Q850, Q700, DVG200, DVG850, DVG925, PE, PWAT, WS500, WS700, WS925) are selected via SHAP analysis to represent dynamics, moisture, and wind fields.

Terrain data : static elevation encodes orographic effects on rainfall distribution.

Network Architecture

DeepFusion adopts an improved U‑Net backbone with three independent encoder branches—one for radar/precipitation, one for NWP, and one for terrain. Each encoder contains multi‑scale feature extractors that progressively down‑sample to capture both fine‑grained convective details and large‑scale background patterns.

The core idea is independent encoding + hierarchical fusion : features from each modality are first processed separately, then fused at matching scales.

Fusion Mechanism

Two levels of fusion are employed:

Intra‑source fusion : variables within the same source (e.g., NWP divergence, wind speed, precipitable water) are concatenated on the channel dimension and processed by a fusion module.

Cross‑source fusion : feature maps from different sources are concatenated at identical spatial scales and passed through a channel‑attention block that learns adaptive weights for each channel.

The attention‑based module enables the network to emphasize the most predictive features for the current forecast.

Decoder

The decoder mirrors the encoder with symmetric up‑sampling via bilinear interpolation, residual blocks, and attention modules. Skip connections merge encoder features at each scale, preserving high‑resolution details. The final output is a 1 km × 1 km precipitation field for 20 future time steps (6‑minute interval).

Training Strategy

Data preprocessing : quality control, outlier removal, spatio‑temporal interpolation; Min‑Max scaling for radar/precipitation, Z‑score for NWP; log‑transform of precipitation labels to mitigate long‑tail distribution.

Data augmentation : random rotations and flips to increase spatial diversity; weighted sampling to raise the proportion of strong‑rain events.

Optimization :

Optimizer: AdamW, initial LR = 1e‑4

LR schedule: 1000‑step warmup followed by cosine decay

Early stopping on validation loss

EMA of model weights for stability

Loss: weighted MSE + λ·(1 ‑ SSIM)

Experimental Results

Ablation of NWP Variables

Three configurations were evaluated on 5 % of the dataset:

exp_nwp0 : only radar and historical precipitation.

exp_nwp1 : NWP variables concatenated after a single convolution layer.

exp_nwp2 : NWP processed by an independent encoder before fusion (the DeepFusion scheme).

Results show that adding NWP variables markedly improves forecast skill, and the independent‑encoding scheme (exp_nwp2) outperforms direct concatenation (exp_nwp1), especially in reproducing the morphology of strong‑rain cells.

Case Study

A representative convective event demonstrates that DeepFusion accurately predicts the movement direction, speed, intensity evolution, and spatial distribution of the rainfall system, with SSIM‑guided loss preserving coherent precipitation patterns.

Conclusion and Future Work

DeepFusion achieves high‑resolution (1 km) and high‑frequency (6 min) 0‑2 hour forecasts with strong accuracy, demonstrating the value of multi‑modal deep fusion and attention mechanisms for short‑term precipitation prediction.

Future directions include probabilistic forecasting to quantify uncertainty, extending the horizon to 2‑6 hours, and incorporating additional observations such as satellite imagery and lightning detection.