Cutting Data Dashboard Development Time from Days to Hours: Xianyu’s 3‑Layer Serverless Solution

Background

Since its founding in 2014, Xianyu has grown to tens of millions of daily active users. The rapid growth exposed limitations in its data‑analysis process, which relied on ad‑hoc BI resources, slow SQL queries, and high front‑end/back‑end integration costs. The team identified four main pain points: limited BI capacity, slow SQL performance, costly integration, and complex data types that hinder insight discovery.

Current Workflow

The existing analysis pipeline consists of three stages: SQL development, application development, and data visualization/report generation. Completing a full cycle typically takes five days or more.

SQL Development

BI engineers write SQL from scratch for each request, despite many requests sharing similar logic. This lack of reuse leads to low efficiency. The proposed approach abstracts SQL into reusable "atomic SQL" units. Front‑end components specify assembly rules, and a composition layer stitches atoms into final queries, dramatically reducing duplicate work. Additionally, queries on billions of rows currently take tens of minutes; the goal is to achieve second‑level responses using an analytical database service.

Application Development

Backend developers use Java to build APIs and data aggregation services. The team suggests moving this layer to a serverless FaaS platform, leveraging the principle that anything implementable in JavaScript can eventually be written in JavaScript.

Data Visualization

Front‑end engineers render dashboards based on data supplied by the backend. Displaying dozens or hundreds of metric curves simultaneously creates cluttered charts, making it hard to spot valuable trends. The solution aims to extract and highlight statistically significant data points, presenting them intuitively.

Technical Solution

The architecture is divided into three layers:

Data Layer

Service Layer (FaaS)

Presentation Layer

The presentation layer sends a DSL (Domain‑Specific Language) description to the FaaS layer. The FaaS layer assembles SQL, executes it against the data layer, processes results, and returns formatted data for visualization.

Data Layer

Provides business‑oriented data and manages underlying storage. Data resides in two places: offline tables (populated by scheduled ETL jobs) and an analytical database that supports second‑level query performance on billions of rows.

Service Layer (FaaS)

Built on the internal Midway FaaS framework, it offers BaaS primitives such as database access, permission checks, logging, and a knowledge‑base service. Key capabilities include:

Data pipeline pattern : A modular pipeline that separates data acquisition, processing, and visualization, enabling flexible, plug‑in components.

Knowledge base : Automatic discovery and semantic annotation of underlying datasets, forming the foundation for downstream processing.

SQL generator : Transforms atomic SQL definitions into final query strings via dynamic compilation and AST parsing, merging duplicate atoms to produce optimized SQL.

Data processing : Converts raw query results into a standardized format for the front‑end.

Presentation Layer

Offers a dashboard with intelligent chart rendering. Two major features:

Custom chart types that emphasize business‑critical fluctuations, encapsulated in a reusable chart library.

Algorithmic extraction (e.g., significance detection) that highlights valuable data points while de‑emphasizing noise.

Implementation Details

Front‑end inputs a DSL that describes how to assemble SQL. The DSL is parsed into sub‑DSLs, each governing a portion of the query. Dynamic compilation produces an intermediate representation, which is then parsed into an AST to identify and merge identical atomic SQL fragments. The final SQL runs against the analytical database, returning results in milliseconds (e.g., 1.7 billion rows queried in 1062 ms).

After query execution, a semantic layer converts numeric codes (e.g., 0/1) into human‑readable labels using the knowledge base, reducing cognitive load for analysts.

Results

The new system delivers:

Flexible SQL assembly with intelligent chart rendering.

Development effort reduced from ~5 person‑days to 0.5 person‑days per dashboard.

Real‑time computation with second‑level response times (1.7 billion rows in ~1 second).

Developers only need to define the DSL; the dashboard renders automatically. Users can interactively customize dimensions, filters, and drill‑down paths, receiving instant feedback.

Future Plans

The team aims to evolve the platform toward low‑code/no‑code capabilities, allowing non‑technical users to drag‑and‑drop data sources, processing steps, and visualizations to build dashboards, while continuing to explore front‑end intelligence enhancements.