How G7 EasyFlow’s Low‑Code Data Sync Platform Boosts Efficiency and Reliability

1. Introduction

G7 EasyFlow faces extensive demands from customers to integrate with industry, large carrier, and regional government transportation regulatory platforms, which require strict data exchange protocols. Failure to comply can lead to loss of transport qualifications or orders, creating a massive need for data collaboration and system integration.

After years of custom development, rising R&D and server costs, and inconsistent service quality, the Chengdu R&D team built a full‑stack, low‑code data synchronization platform in 2019 to centralize and streamline these services. Over six years of iteration, it has become a company‑wide low‑code development platform for cross‑team delivery.

2. The Platform’s Core Questions

2.1 What is the platform?

The G7 EasyFlow Data Sync Platform converts and synchronizes data between distributed systems, ensuring eventual consistency and providing a low‑code environment for R&D delivery.

2.2 Current status

Since its launch, the platform has completed 77 projects, handled over 400 system integration requests, and processes 4.35 billion messages daily, making it one of the company’s largest data‑processing platforms.

2.3 Value and goals

It aims to make inter‑system data synchronization effortless, delivering high‑performance, highly available services with the "three‑high" characteristics (high availability, high concurrency, high consistency).

Centralized management of multi‑system data conversion with industry‑leading best practices.

Visual configuration to improve development efficiency and reduce ongoing operational costs.

Long‑term stability and guaranteed data consistency for external data quality.

3. Platform Advantages

3.1 Long‑term stability

Operating for over five years without any P3‑level incidents, with total downtime under 30 minutes as of 2025‑04‑30.

3.2 Core features

Low‑code : Embedded script engine with AI‑assisted code generation enables zero‑code rapid development.

High availability : Multi‑node, multi‑instance consumption and production, horizontal scaling, with up to 8 nodes and 512 consumer instances.

High concurrency : Parallel consumption and production, supporting over 100 k TPS in practice.

Rate limiting : Configurable consumption rates to smooth traffic peaks.

Eventual consistency : Configurable fault‑tolerance windows ensure data convergence.

4. Architecture

The platform uses a simple technology stack:

Backend : PHP (Web API), Yafrk (custom framework), Golang (Agent), Gin.

Frontend : JavaScript, Vue2, Ant‑Design‑Vue.

Scripting engine : gojs (pure Go implementation of ES5.1).

4.1 Overall architecture

Data flows from source to target through a series of coordinated services, as illustrated in the architecture diagram.

4.2 Coroutine model

Each task spawns independent coroutines: a main coroutine creates consumer, producer, and auxiliary coroutines. Producer coroutines generate worker pools for parallel execution of push logic.

4.3 Data flow

The shortest path from source to target is depicted in the data‑flow diagram.

4.4 Final consistency mechanisms

Four layers protect against data loss:

Retry queue : In‑memory data is persisted to Redis on node shutdown or task failure, and automatically retried on recovery.

Failure queue : Errors write messages to a Redis failure queue, which periodically moves data back to the retry queue.

Auto‑pause/resume : When failure queue thresholds are exceeded, consumption pauses and resumes once the backlog drops.

Offset reset : After exceeding retry limits, messages may be discarded; offsets can be reset if downstream services support idempotency.

5. Standards

5.1 Message schema

Tasks require predefined message schemas, enabling visual configuration and consistent data contracts across systems.

5.2 Data forwarding protocols

Supported protocols include HTTP, HTTPS, Inward, Vega, RocketMQ, and Kafka; Inward and Vega are internal private protocols based on Eureka.

5.3 Message processing stages

The platform defines eight optional stages: filtering, field mapping, request signing, request batching, message push, response validation, cleanup, and global exception handling. Each plugin receives STDIN and returns STDOUT.

5.4 Embedded script engine

JavaScript (ECMAScript 5.1) scripts can be embedded at any stage, providing flexibility and extensibility.

5.5 Distributed deployment

Tasks can be deployed across multiple nodes and instances to handle high‑concurrency scenarios, with configuration parameters such as scheduling strategy, node count, instance count, QPS, and ART.

5.6 Logging standards

Logs follow the TMG standard, using a trace_id (message ID) for end‑to‑end traceability in ELK, with configurable log levels per task.

6. Core Functional Preview

6.1 Supported scenarios

KAFKA queue consumption (standard and IoT rule‑engine modes).

RocketMQ queue consumption.

HTTP timed polling for full or incremental sync.

HTTP API orchestration for cross‑system real‑time synchronization.

6.2 Custom message schema

Users can browse existing schemas or create new ones for Kafka, API requests, etc.

6.3 Project and task management

Projects group related tasks; tasks are created, configured, synchronized to agents, started, and monitored via the UI.

6.4 AI‑assisted code generation

All plugins and JS components support AI‑assisted writing and code completion, allowing developers to generate functional code from a single sentence description, often achieving zero‑modification deployment.

Author: Product Technology Division – Xu Dianyang