Design and Implementation of a Data Service Middle Platform for Scalable Data SaaS

As business demands for data grow, the article introduces the concept of data service SaaS, aiming to transform raw data into reusable services such as APIs, RPC, and data files, and outlines the pain points that motivate the creation of a standardized, middle‑platform data service solution.

The proposed platform architecture consists of a data construction layer, a data query layer, service interfaces, a service gateway, and supporting modules for data standards, security, and operations monitoring. The data construction layer abstracts warehouse tables into business‑oriented models, supporting model definition, acceleration, and API generation.

Model definition covers various dimensional modeling approaches (single, star, snowflake, constellation) to meet diverse analytical scenarios. Model acceleration offers two strategies: detail acceleration (mirroring data from cold to hot engines for multidimensional queries) and pre‑calculation acceleration (aggregating data in advance for high‑performance queries), with recommended engine combinations for online, near‑online, OLAP, and offline use cases.

API construction standardizes parameters (API ID, name, method, path, request/response fields, QPS, etc.) and supports three configuration methods:

Custom SQL using MyBatis‑style dynamic tags (if, foreach, where). Example: select a.field1 AS alias_1, a.field2 as alias_2, a.field3 as alias_3, b.field1 as alias_4 from fact_table a left outer join table_dim b on a.id = b.id <where> a.field = ${ input_1,type = number } <if test='input_2 !=null'> and b.field = ${ input_2,type = number } </if> </where>

Visual model‑based construction without writing SQL.

Metric‑dimension construction that auto‑selects suitable models based on defined metrics and dimensions.

Service gateway functions include authentication (appKey/secret), rate limiting based on QPS estimates, and monitoring of request counts, success rates, and error types. Service interfaces support synchronous, asynchronous, DSL, template, and raw SQL queries, with protobuf definitions such as: message OpenApiReq { OsHeader osHeader = 1; repeated OperatorVo filters = 2; repeated string metrics = 3; repeated string dims = 4; repeated string orderFields = 5; PageVo pageVo = 6; repeated OperatorVo metricFilters = 7; } and message SqlQueryReq { OsHeader osHeader = 1; repeated OperatorVo filters = 2; } .

The article also describes common solutions for unified metric definitions, end‑to‑end traceability, cost reduction through model and API reuse, and high availability measures such as service isolation, active‑active multi‑region deployment, and layered caching (local and distributed) with version management.

Since its inception, the platform has delivered over 500 APIs with tens of thousands of QPS, reduced API creation time from five days to under one, and cut production costs by roughly 18%, demonstrating significant efficiency gains for both C‑end and B‑end services.

Future work focuses on improving stability, introducing intelligent automation to reduce manual metadata registration, and establishing long‑term governance mechanisms to maintain API health and performance.