Big Data 12 min read

How Litefuse’s New Single‑Process Mode Lets an Agent Observability Platform Run in 25 seconds

Litefuse open‑sources a single‑process, sub‑400 MB binary that deploys an Agent observability and evaluation platform in about 25 seconds, explains why Docker‑free deployment matters, and details how Apache Doris’s inverted index, VARIANT JSON type, and compute‑storage separation address the massive, long‑text, semi‑structured traces that differentiate Agent monitoring from traditional observability.

DataFunSummit
DataFunSummit
DataFunSummit
How Litefuse’s New Single‑Process Mode Lets an Agent Observability Platform Run in 25 seconds

Single‑process mode

Running the platform requires a single command: curl -fsSL https://litefuse.ai/install.sh | sh The video demonstration shows a 358 MB download completed in 12 seconds, extraction in 11 seconds, and full deployment in about 25 seconds.

In the same environment, the Docker‑based Langfuse deployment takes 2 minutes 18 seconds, roughly 5.5× slower.

Architecture of the single‑process mode

A single binary under 400 MB bundles the Litefuse program, an embedded Node.js runtime, a JVM, PGlite (embedded PostgreSQL), and DorisLite (embedded Apache Doris).

No external runtime dependencies are required; the binary runs as one OS process, with PGlite and DorisLite loaded as libraries.

All components start together, eliminating Docker orchestration and multi‑process management.

Supports macOS (Apple silicon) and Linux (x64).

Why Apache Doris as the storage‑analysis engine

Agent traces differ from traditional observability data in four ways: MB‑scale long text, multi‑day GB‑scale traces, heavy semi‑structured JSON, and an order‑of‑magnitude increase in total data volume.

Inverted index (supported since 2023) accelerates full‑text search up to 10× and provides operators such as MATCH, MATCH_ALL, MATCH_PHRASE, plus multilingual tokenizers.

Delayed materialization for TOPN queries reads only the timestamp during sorting and fetches full rows for the final N results, reducing I/O, CPU and memory consumption.

Bucketed, sorted storage hashes data by trace‑id, distributes buckets across servers, and sorts within each file, enabling fast range scans for ultra‑long traces.

VARIANT JSON type automatically extracts JSON fields into columnar sub‑columns, compresses them, and allows column‑pruned reads, delivering up to a 10× speedup for JSON‑heavy queries.

Columnar storage & compute‑storage separation with ZSTD compression reduces disk usage; storing a single replica in object storage cuts storage space by >50 % and overall cost by 75‑88 % compared with multi‑replica local disks.

ClickHouse, the engine behind Langfuse, does not support compute‑storage separation and therefore requires multiple local replicas, leading to higher storage cost.

Open‑source release

Litefuse is released under the MIT license at https://github.com/litefuse/litefuse. The installation script is hosted at https://litefuse.ai/install.sh.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

cost optimizationinverted indexApache DorisAgent observabilityLitefusesingle-process deploymentVARIANT JSON
DataFunSummit
Written by

DataFunSummit

Official account of the DataFun community, dedicated to sharing big data and AI industry summit news and speaker talks, with regular downloadable resource packs.

0 followers
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.