Apache Arrow 1.0.0 Released with New Columnar Format Features

Apache Arrow 1.0.0 is now released. The official statement says this version is the 18th major release, marking the transition of the columnar format to binary stability and the Arrow libraries to semantic versioning.

Apache Arrow is a top‑level Apache Foundation project aiming to serve as a cross‑platform data layer to accelerate big‑data analytics. It defines a set of in‑memory flat and hierarchical data representations, provides language bindings for structured operations, and offers low‑overhead streaming, batch messaging, zero‑copy inter‑process communication (IPC), and vectorized memory analytics libraries.

Some changes to the columnar format include:

Metadata version has been changed to new version V5, indicating an incompatible change in the buffer layout of Union types.

Dictionary indices now may be unsigned integers rather than only signed integers; due to poor Java support, UInt64 is still discouraged.

A new "Feature" enum has been added.

Optional buffer compression using LZ4 or ZStandard has been added to the IPC format.

Decimal types gained an optional "bitWidth" field, defaulting to 128; the team says this will enable future support for other decimal widths such as 32‑bit and 64‑bit.

Validity bitmap buffers have been removed.

Additionally, the team expanded integration tests to cover extended types and nested dictionaries, and updated support for C++, Java, Python, R, Ruby, C GLib, and Rust.