Designing a Sparse‑Matrix Based Format for Radar dBZ Data (smdbz)

dBZ data characteristics

Radar reflectivity (dBZ) is used for short‑term precipitation forecasts. Values follow the national charting standard: 14 levels from 5 to 75 dBZ in steps of 5 plus a zero level, i.e. 15 grades that can be encoded as integers 0‑14. Because 15 < 2⁴, each grade fits in 4 bits (half a byte).

Sparsity of radar grids

Statistical analysis of global radar mosaics shows that in most frames the non‑zero echo occupies <5 % of the grid, i.e. >95 % of cells are zero. This matches the definition of a sparse matrix.

Sparse‑matrix compression (CSR)

Compressed Sparse Row (CSR) stores only non‑zero entries with three arrays: val – flattened list of non‑zero values. colind – column index for each value. rowptr – start index of each row in val (length = rows + 1).

Figure (Elafrou et al., 2015) illustrates the layout.

Extracting CSR parameters with SciPy

import numpy as np
from scipy import sparse

array = np.array([
    [7.5, 2.9, 2.8, 2.7, 0, 0],
    [6.8, 5.7, 3.8, 0, 0, 0],
    [2.4, 6.2, 3.2, 0, 0, 0],
    [9.7, 0, 0, 2.3, 0, 0],
    [0, 0, 0, 0, 5.8, 5.0],
    [0, 0, 0, 0, 6.6, 8.1]
])

sparse_array = sparse.csr_matrix(array)
rowptr = sparse_array.indptr
colind = sparse_array.indices
values = sparse_array.data
print("rowptr", rowptr)
print("colind", colind)
print("value", values)

rowptr [0 4 7 10 12 14 16]
colind [0 1 2 3 0 1 2 0 1 2 0 3 4 5 4 5]
value [7.5 2.9 2.8 2.7 6.8 5.7 3.8 2.4 6.2 3.2 9.7 2.3 5.8 5.0 6.6 8.1]

Random access works like a dense ndarray:

print(sparse_array[2, 1])  # 6.2

Custom read‑only CSRMatrix for 3‑D time stacks

SciPy’s csr_matrix handles only 2‑D matrices. To store a time‑stacked (y, x, t) radar field, a lightweight read‑only class is defined:

class CSRMatrix:
    def __init__(self, data: list, indices: list, indptr: list, shape: tuple):
        self.data = data
        self.indices = indices
        self.indptr = indptr
        self.shape = shape

    def __get(self, row: int, col: int):
        if row >= self.shape[0] or col >= self.shape[1]:
            raise IndexError("Index out of range")
        start = self.indptr[row]
        end = self.indptr[row + 1]
        for i in range(start, end):
            if self.indices[i] == col:
                return self.data[i]
        return 0

    def __getitem__(self, key):
        row, col = key
        return self.__get(row, col)

Instantiating with the CSR parameters above:

csrm = CSRMatrix(values, colind, rowptr, array.shape)
print(csrm[5, 4])  # 6.6

The class also accepts nested lists as data, enabling each grid point to hold a list of values for different time steps.

values = [
    [7.5, 0], [2.9, 0], [2.8, 0], [2.7, 0],
    [6.8, 0], [5.7, 0], [3.8, 0], [2.4, 0],
    [6.2, 0], [3.2, 0], [9.7, 0], [2.3, 0],
    [5.8, 0], [5.0, 0], [6.6, 0], [8.1, 0]
]
csrm = CSRMatrix(values, colind, rowptr, array.shape)
print(csrm[5, 4])  # [6.6, 0]

Handling multi‑frame matrices

In a (y, x, t) array a coordinate is zero only if *all* time‑step values are zero; otherwise it is non‑zero. The union of non‑zero coordinates across frames yields a 2‑D mask from which rowptr and colind are derived. The nested val array is built manually.

Bit‑packing of dBZ grades

Because only 15 grades exist, two grades can be packed into one byte:

# pack grade 5 (0b0101) and grade 8 (0b1000)
packed = (5 << 4) | 8   # 0b01011000 == 88
# unpack
high = packed >> 4      # 5
low  = packed & 0x0F    # 8

smdbz binary layout

The file consists of a fixed‑size header followed by three variable‑size sections:

Header (5 int): frame length N₁, matrix height, matrix width, indices length N₂, indptr length N₃.

Body: indices – N₂ int values. indptr – N₃ int values.

Compressed value – unsigned char array of length N₁//2 + N₁%2.

Reading a file is implemented in the open‑source repository https://github.com/caiyunapp/smdbz.

Reading a .smdbz file

from smdbz import read_smdbz
smdbz = read_smdbz("./global.smdbz")
print(smdbz[2041, 1529])

Result (future 2‑hour forecast in 5‑minute steps):

[5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 5, 5, 4, 4, 0, 0, 4]

Advantages and limitations

Random access requires only three lines of Python, avoiding the need to load and stitch dozens of PNG files.

File‑size comparison (uncompressed): PNG ≈ 6.3 MiB, smdbz ≈ 18 MiB, npy ≈ 550 MiB.

After zip or bzip2 compression smdbz becomes the smallest (≈2.4 MiB) while npy remains larger.

CSR overhead makes the format inefficient for dense matrices; it also lacks support for vectorised operations and AI‑oriented scientific computing.

Size comparison (uncompressed and after compression)

PNG – 6.3 MiB (zip 6.0 MiB, bzip2 6.0 MiB)

smdbz – 18 MiB (zip 2.6 MiB, bzip2 2.4 MiB)

npy – 550 MiB (zip 4.4 MiB, bzip2 2.5 MiB)