Design and Implementation of a Data Consistency Engine for Advertising Billing Systems

1. Project Background

Advertising retrieval systems generate a large portion of company revenue, and the billing subsystem is a critical low‑level component responsible for anti‑fraud, fee calculation, discount deduction, and actual charge deduction, involving data consistency across billing, marketing, payment, and budget systems.

Because billing methods such as CPT, CPA, CPC differ and ad clicks are non‑replayable, a robust data‑consistency engine is essential.

2. Consistency Solution Selection

Payment systems can either hold a payment license and interact directly with banks, or invoke third‑party payment services. The first scenario demands strong guarantees like TCC (two‑phase commit) for strict consistency, while the second allows lighter solutions such as local message tables or transactional messages.

Our project falls into the second scenario, prompting the evaluation of lightweight approaches.

Transactional Message

Implementation is lightweight with low transformation cost, suitable for scenarios with modest real‑time requirements.

However, our in‑house message middleware does not support it, and the approach can lead to tightly coupled upstream and downstream services.

TCC Mode

TCC requires each participating system to implement try/commit/cancel interfaces, maintain primary and branch transaction tables, and provide idempotent tables to avoid dangling deductions.

It is suitable for user‑direct resource deduction scenarios but is cumbersome for ad click streams where clicks cannot be replayed, requiring both rollback and retry capabilities.

Saga Mode

We ultimately adopted a saga‑style state‑machine engine with compensation logic, which is friendly to legacy system refactoring, allows flexible state storage, and supports configurable retry, rollback, and asynchronous compensation strategies.

3. Data Consistency Engine Overview

Engine Architecture Diagram

(Image omitted)

Structural Components

State Machine : orchestrates node execution order and other execution features.

Node : business logic such as CPC billing, including pre‑check, price adjustment, coupon, and pmc deduction.

Compensation Logic : per‑node fallback actions for error recovery.

Hook Functions : allow custom operations before and after engine execution.

Scheduled Tasks : trigger points for abnormal data recovery.

Additional Features

Compensation Mode : configurable retry or rollback.

Compensation Timeliness : real‑time or asynchronous delay.

Retry Count & Time Decay : customizable retry sequences.

State Machine Configuration Example

{
  "name": "xxxx_xxxx_xxxx",
  "comment": "cpc计费状态机",
  "firstNodeName": "check",
  "nodes": {
    "check": {
      "nextNodeName": "land",
      "preNodeName": "",
      "skipRecover": true
    },
    "land": {
      "nextNodeName": "antiFraud",
      "preNodeName": "check",
      "skipRecover": false
    },
    "antiFraud": {
      "nextNodeName": "realPrice",
      "preNodeName": "land",
      "skipRecover": false
    },
    "...": {"..."}
  },
  "retryCount": "4",
  "timeDecaySeries": ["1","3","5","10"],
  "recoverType": "Retry",
  "compensateTimeliness": "ASYNC"
}

Engine Initialization

DTConfig.builder()
    .setAppName("billing") // configure app name
    .setLogStoreStrategy(StoreStrategyEnum.DEFAULT_STORE) // storage strategy
    .setRedisConfig(redisConfigPath) // Redis config
    .setDBTableConfig(mysqlConfigPath) // MySQL config
    .setZKConfig(configPath) // ZK config for gray release
    .setStateMachinePath(stateMachinePath) // state‑machine config path
    .setNegligibleErrorCode(BillingDTConstants.serious_error_code_str) // critical error codes
    .build();

Engine Invocation

// State machine name for this call
String stateMachineName = "ecpm_state_machine";
// Obtain engine instance
DTBizEngine dtBizEngine = new SagaDTBizEngine();
// Execute request
DTResponse response = dtBizEngine.start(new DTEngineRequest(bizType, bizId, stateMachineName, originContext));
System.out.println(response.getData());

Asynchronous Execution (Reference to Dubbo)

Main Thread

try {
    // Create DTFuture with request and timeout
    DTFuture mFuture = new DTFuture(request, 1000);
    request.getExtendField().put("MY_KEY", mFuture);
    // Async thread call
    EcpmEventBus.getInstance().post(new EcpmBillingEvent(request));
    // Wait with timeout (ms)
    Object future = mFuture.get(1000);
    // do something
} catch (Exception e) {
    // handle exception
}

Execution Thread

// Retrieve future from request
DTFuture future = (DTFuture) request.getExtendField().get("MY_KEY");
DTFuture.received(future.getId(), response);

4. Conclusion

This article introduced the background and solution‑selection process for a data‑consistency project and presented a reference saga‑style state‑machine implementation as the foundation for a distributed‑transaction engine.

It serves as the first part of a series on distributed transaction systems; subsequent articles will detail the design and usage of each engine module.