Skip to content
JSONL vs JSON: When Line-Delimited JSON Wins for Logs and ML Data

JSONL vs JSON: When Line-Delimited JSON Wins for Logs and ML Data

This article compares JSONL (JSON Lines) with regular JSON, explaining how line-delimited records enable memory-efficient streaming for logs, machine learning datasets, and ETL pipelines, while detailing tool support in pandas, Spark, and Hugging Face, and clarifying when to stick with standard JSON.

Data STUDIO
Data STUDIO
Data STUDIO

01 Difference Lies in Record Boundaries, Not JSON Syntax

Regular JSON has a single root value (object or array). Even with millions of records, the entire array remains one JSON document. JSONL splits records into one line per record, allowing programs to consume already-written portions without waiting for all records to arrive. Producers can append new data by simply writing another line.

{"id": 1, "event": "login", "user": "Lin"}
{"id": 2, "event": "purchase", "user": "Mei"}
{"id": 3, "event": "logout", "user": "Lin"}

Equivalent JSON array:

[
  {"id": 1, "event": "login", "user": "Lin"},
  {"id": 2, "event": "purchase", "user": "Mei"},
  {"id": 3, "event": "logout", "user": "Lin"}
]

JSONL lets each line be an independent, parsable JSON value. Logs, datasets, and bulk exports often need this granularity.

02 What Line-by-Line Reading Saves in Memory

Switching to JSONL does not automatically reduce memory; the key is how the program processes data. A 50 MB JSON array loaded entirely may peak well above file size due to object and container overhead. With JSONL, you can read one line at a time:

import json

event = {"id": 4, "event": "login", "user": "Chen"}
with open("events.jsonl", "a", encoding="utf-8") as f:
    f.write(json.dumps(event, ensure_ascii=False) + "
")

The "a" mode appends. Each new event serializes to one JSON line plus newline, letting the file grow continuously — more suitable for continuous event recording than maintaining a full JSON array.

JSON Lines format requires UTF-8, each line a valid JSON value, blank lines not counted as records; typically \n newline, \r\n also accepted. A trailing newline after the last record simplifies further appends or concatenation.

A line need not be a JSON object — any valid JSON value (string, number) is allowed, though objects with id, timestamp, event fields are most common.

Using splitlines() with json.loads() works for ad-hoc scripts, but when multiple services and teams share files, explicit agreements on encoding, blank lines, malformed lines, and newline rules reduce coordination overhead. The public JSON Lines / NDJSON conventions and tooling options help.

Example line-by-line processing that keeps memory low:

import json

with open("events.jsonl", encoding="utf-8") as f:
    for line_number, line in enumerate(f, start=1):
        if not line.strip():
            raise ValueError(f"Line {line_number} is empty")
        record = json.loads(line)
        print(record["event"])

Blank lines are treated as errors here; adjust if skipping is needed, but producer and consumer should agree.

If you instead accumulate records with records.append(record), memory grows with data volume. Downstream full sorts or complex joins may also need extra space. Extremely large single records likewise demand sufficient memory.

Regular JSON has incremental parsers too: Python's ijson, Node.js's stream-json can parse large JSON documents stepwise without building the full object tree. Use them when upstream provides a complex document with nested arrays that must stay intact. JSONL suits data that can be split into independent records from the start. Both approaches are worth knowing; choice depends on data structure and downstream needs.

03 Tool Support Must Be Verified Per Tool

With pandas, reading JSONL is straightforward:

import pandas as pd

for chunk in pd.read_json(
    "events.jsonl",
    lines=True,
    chunksize=100_000,
):
    process(chunk)

Exporting millions of order records, cleaning fields, dropping invalid rows, and batch-writing to a database fits this pattern. The 100_000 batch size is illustrative; actual size depends on record field count, transformation steps, and machine memory. Pandas docs note chunksize requires lines=True; without chunking, data loads entirely.

Spark's JSON data source defaults to JSON Lines processing. If upstream supplies a pretty-printed multi-line JSON document, you must enable multiLine. This mismatch is a common pitfall: a colleague saves a formatted JSON array across dozens of lines, Spark's default reader yields unexpected results because input structure and read options don't align.

Hugging Face Datasets also supports JSON/JSONL with streaming=True for iterative access. For 100k text classification samples with text and label fields, line-by-line storage enables incremental loading, cleaning, and transformation. The training framework's own format requirements still apply.

Streaming does not equal low memory in all cases; later large-scale shuffle, caching, or preprocessing can change resource usage.

When the same file goes to databases, ETL services, or other language readers, check each tool's supported options. Some default to line-by-line, others require explicit flags; blank-line and error-line handling may differ.

04 Common JSONL Use Cases

First: logs and events. Each request, login, purchase, or logout produces a record. Such data is naturally independent and appendable. To count today's login events, scan lines and update a counter. Tail-reading new lines from the file end is also simple. Production still needs timestamp handling, field versioning, log rotation, and bad-line recovery, but line-oriented files make collection, export, and debugging more direct.

Second: machine learning training data and evaluation results. For 100k QA samples with question, reference answer, and label, save each sample as one JSON line. Later, check missing fields, filter anomalies, or batch-convert to the training framework's structure. Evaluation fits too: a run with tens of thousands of questions writes one result per completed item. If the run stops mid-way, already-written lines remain readable. Resume logic and deduplication rely on sample IDs and external state; JSONL just provides a convenient save format.

Third: data migration and ETL. Bulk-export customer data from a legacy system, clean it, output JSONL, then let downstream map fields line by line. A malformed record is easily located by line number, allowing fail-fast, error logging, or continue-on-error decisions. Field names, types, and null conventions must still be agreed upfront — JSONL doesn't decide whether created_at is a string or timestamp, or how to represent a missing phone number.

Fourth: big data intermediate files. Raw events as JSONL are easy to inspect and exchange; later, convert to Parquet for large-scale columnar queries. Spark can leverage per-record boundaries for reading, but columnar scan and analysis performance depend on the actual workload. No single file format needs to cover every stage.

Message queues and streaming APIs deserve separate mention because they're often conflated with JSONL. Message queues already have message boundaries; a message payload can be a JSON object. JSONL typically enters the picture when exporting messages, archiving logs, or exchanging line-delimited text files. SSE (Server-Sent Events) has its own event-line and delimiter rules. An SSE data field may contain JSON, but the whole SSE stream cannot be parsed as JSONL (one JSON value per line). When seeing "streaming JSON", verify how the interface delineates messages before choosing a parser.

05 When to Stick with Regular JSON

Some scenarios work fine with JSON and need no change. A REST API returning user profile with nested personal info, permissions, and settings: the caller expects a complete response, and a standard JSON parser handles it directly. Switching to JSONL would require a new line-by-line reading protocol.

Config files are similar. At startup, an app reads database connections, log levels, and feature flags — usually a small file where JSON's object hierarchy is clear. If comments or trailing commas are desired, consider JSONC or JSON5, but confirm toolchain support.

A frontend displaying dozens of products: a JSON array is natural. When data volume is controllable and the consumer always needs the full set, there's no reason to change formats for the sake of "streaming".

Even with the same 50 MB file, some small scripts struggle while services handle it easily. More important than file size are: will this data keep growing? Can a single record be consumed independently? Does downstream require the full dataset? Are incremental JSON parsers available? Answer these first instead of setting a fixed capacity threshold between JSON and JSONL.

06 JSONL vs NDJSON: Similar but Conventions Differ Beyond Extension

NDJSON (Newline-Delimited JSON) is very close to JSON Lines — both separate JSON texts by newline. Both have public specs, but common extensions and media types differ. JSON Lines often uses .jsonl and mentions application/jsonl (not yet standardized). NDJSON's spec recommends .ndjson and application/x-ndjson.

JSON Lines allows any valid JSON value per line. NDJSON requires each JSON text to conform to RFC 8259, with internal newlines and carriage returns escaped. Many tools treat them as similar input formats, but whether each line must be an object, whether blank lines are skipped, and whether parsing continues or stops on error depend on the specific implementation.

Example: System A gives you events.jsonl, System B's importer says "supports NDJSON". Test a few real samples for encoding, newlines, and error handling before processing multi-gigabyte production files — don't rely on the extension alone.

07 Why This Format Is Often Overlooked

JSONL is so simple it barely feels like something to learn: one JSON per line, process by line. In throwaway scripts, defining newlines and calling json.loads() works. But once a file crosses languages and services, small details suddenly matter: an extra blank line upstream — does downstream error? Concatenating files where the first lacks a trailing newline merges two records into one.

Its practical value shines in data handoffs: clear record boundaries, existing tooling can read it, and issues are easy to locate.

Of course, if data is a complex single JSON document, use an incremental parser; if the primary goal is columnar analysis, consider Parquet; if the interface defines complete JSON requests or responses, follow the interface spec. Choosing a format based on the actual workflow is often more effective than debating which format is more advanced.

08 Closing Thoughts

Next time you get a large JSON file, confirm three things with your team: can records be processed independently? Will the file keep growing via appends? Can downstream programs handle one record or one batch at a time and then release memory?

If all three are yes, JSONL is worth a try. Logs, training samples, evaluation results, ETL intermediate files, and bulk exports are typical scenarios.

If the file represents a complete hierarchical document, or the consumer genuinely needs the whole thing at once, regular JSON may be simpler. If memory issues arise solely from file size, don't forget incremental JSON parsers as an alternative.

Ultimately, it's about how data is produced, how it's handed to the next step, and whether code truly consumes by line or by batch. Pick the right record boundary, and the downstream processing flow usually becomes much simpler.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

ETLstreaming dataApache SparkPandaslog processingmemory efficiencydata serializationHugging FaceNDJSONincremental parsingmachine learning datasetsJSON LinesJSONL
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.