How ByConity Powers Real‑Time Telecom Data Analytics: A Deep Dive

Challenges of Real‑Time Data Analysis for Telecom Operators

Haijing Technology, founded in 2023, serves telecom operators worldwide and increasingly needs small, independent deployments. Their legacy Hadoop stack, even with Flink, struggled with real‑time incremental aggregation, query performance (5‑10× slower than modern MPP databases), write bottlenecks, limited cluster scaling, concurrency limits, and low data‑load throughput.

Why Use ByConity

To overcome Hadoop's real‑time shortcomings, they first tried ClickHouse for its strong single‑table aggregation but faced difficulties with complex multi‑table joins, diverse table engines, and distributed key definitions. After ByConity was open‑sourced, it resolved these pain points with a unified cnchMergetree engine, simplified multi‑table joins without manual distribution keys, and seamless elastic scaling.

ByConity Advantages

Unified table engine simplifies development.

Multi‑table join performance dramatically improves, eliminating the need for manual distribution keys.

Separation of compute and storage enables easy horizontal scaling without data reshuffling.

Telecom Real‑Time Analytics Scenario

The workflow aggregates data from OLTP databases, event streams, file storage, and logs into ByConity, enabling ad‑hoc, list, and real‑time analyses.

Operator Real‑Time Analysis Overview

Operator data spans three domains: B (CRM, order center, etc.), O (mobile number activation, resource control), and M (performance, HR). Raw data from each domain—thousands of tables—are ingested, cleaned, and real‑time aggregated into wide tables and complex metric tables, then unified across domains for cross‑domain analytics.

Visualization tools map identical business keys across physical models to create a unified business view, automatically recognizing relationships for downstream metrics and tagging.

Metrics are organized into hierarchical classifications, standardized, and exposed externally.

Hybrid Hadoop‑ByConity Ad‑Hoc Query Solution

A batch‑stream integration merges real‑time Kafka streams (call‑detail records, orders) with hourly batch aggregations in Hadoop. Both streams are consolidated in ByConity, refreshed every five minutes, and visualized on dashboards. Complex calculations are performed upstream, leveraging ByConity’s fast multi‑table joins.

Deep Integration and Packaging of ByConity

Post‑open‑source, ByConity was extensively tested and wrapped into a Warehouse‑as‑a‑Service (WhaleHouse) offering. Key enhancements include:

Physical‑machine deployment to accommodate non‑containerized environments.

Support for arbitrary node counts, allowing single‑node installations.

Visual management of database instances, HDFS, and FoundationDB.

Elastic scaling with dynamic node addition and online start/stop.

One‑click online upgrades via UI.

WhaleHouse serves as the analytics layer, delivering ultra‑fast unified data analysis, real‑time OLAP, massive list queries, federated searches, and time‑series analytics. It supports MySQL and ClickHouse syntax, vectorized execution, and horizontal elastic scaling.

Future Plans

Rapid data migration from Oracle/Greenplum via visual components.

Integrated offline backup and restore capabilities.

Data lake analytics using Hudi and Hive with materialized views.

Real‑time materialized view applications.

Enhanced time‑series analysis leveraging ByConity’s high TPS and aggregation performance.

Through this comprehensive solution, Haijing Technology provides telecom operators with robust, real‑time data measurement, visualization dashboards, and decision‑making capabilities.