Comprehensive Overview of Laiye Technology's Business Intelligence Ecosystem

Laiye Technology is an intelligent automation company serving both B2B and B2C markets, with a strategic focus on B2B. To support its SaaS and on‑premise deployments, the company has built a comprehensive Business Intelligence (BI) system that spans the entire data lifecycle.

Background and Development Stages : The article defines BI as a solution that turns data into quantifiable business value. Laiye’s BI journey is divided into an early development stage—characterized by ad‑hoc log‑based metrics and manual gRPC interfaces—and a growth stage, where a hybrid strategy of open‑source components and public‑cloud services is adopted to meet expanding data‑driven needs.

Data Access : Data is collected through three main mechanisms—data‑point (埋点) logging, front‑end SDKs (Sensors Analytics), and backend logs. Logging follows a low‑intrusion format:

[INFO]2020-12-15 16:34:39 saas_log_statistic={"theme":"staff_operation","timestamp":1608021279460,"operation_id":980064}

. The system emphasizes atomic point logging, idempotency, and the use of business‑meaningful IDs, UUIDs, or generated sequence numbers.

Data Transmission, Transformation, and Loading : Laiye uses Logstore, Kafka, and Pulsar for data transport, preferring Pulsar for its record‑based acknowledgment and multi‑tenant support. Transformation is performed with Logstore scripts, Flink, and Spark, handling standardization, cleaning, and enrichment. Loading targets include MySQL REPLACE statements for idempotent upserts and MongoDB upserts for document stores.

Data Modeling and Storage Layers : The data warehouse follows a multi‑layer architecture: ODS (operational store), DW (with DWD, DWM, DWS layers), APP (application layer), and DIM (dimension tables). Each layer has distinct responsibilities, from raw fact storage to aggregated analytical tables, supporting both real‑time and batch queries. AnalyticDB (ADB) serves as the core real‑time warehouse, with MongoDB handling auxiliary tables.

Statistical Analysis and Real‑Time Metrics : Pre‑defined metrics are generated via a configurable service that aggregates fact tables into high‑level indicator tables. An example JSON configuration is shown below:

{
    "grain_type" : "daily",
    "group_by" : [["account_id","account_id"]],
    "date_field" : ["date_key", "date_key"],
    "time_field" : ["time_key", "time_key"],
    "out_table" : "dw_stats_account_daily",
    "stat_tables" : [
        {
          "source_tbl" : "dw_fact_message",
          "out_field" : "bot_count_active",
          "raw_func" : "count(*)"
        },
        {
          "source_tbl" : "dw_stats_staff_daily",
          "out_field" : "staff_login_number",
          "raw_func" : "sum(login_number)",
          "filter" : "(op_type='LOGIN')"
        }
    ]
}

Ad‑hoc queries are enabled through Redash, which connects to the data warehouse and Logstore, allowing users to write SQL for custom reports. Real‑time indicators are produced by Flink jobs that merge streams from Pulsar and Logstore, generating both detailed event tables and aggregated metrics. Visualization and Product Integration : In the SaaS product, metric tables are exposed via gRPC services for low‑latency consumption. Redash dashboards provide self‑service analytics, data‑driven alerts, and automated reporting. The company also integrates a third‑party multi‑dimensional analysis tool (Sensors Analytics) for user‑behavior analytics. Conclusion and Outlook : While the BI platform has alleviated many early‑stage pain points, challenges remain, such as breaking data silos, deepening end‑user analytics, and further automating operations. Future efforts will focus on enhancing data‑driven decision‑making, product intelligence (e.g., personalization, recommendation), and operational intelligence.