How to Quickly Fix RPC Timeout Data Inconsistency with a Lightweight Mock/Spy Tool

Introduction

The article analyzes data consistency problems in RPC service calls under network timeout conditions, especially when interfaces lack idempotency or idempotency fails. It presents a lightweight mock/spy data correction tool that can mock or spy on service calls, independent of RPC, JSF, WebService, or HTTP, as long as the method can be proxied. The tool is already used in production to handle critical scenarios.

Origin

The idea originated from fault recovery work, aiming to design and develop a data processing efficiency tool for specific scenarios.

Scenario Analysis

In distributed architectures, network communication between applications has three states: success, failure, and timeout.

Success: Request is sent successfully and receives a correct response.

Failure: Request fails to send or the response indicates an operation failure.

Timeout: No response is received within the specified time; the service may have completed successfully or not.

For successful calls, normal processing occurs. For failures, data rollback or compensation can be applied. For timeouts, the caller perceives a timeout while the provider may have completed the operation, making it uncertain whether the data was persisted.

When a timeout occurs, rolling back the caller’s data and attempting to roll back the provider’s data often fails because the provider may still be processing. Delaying the rollback can work, but the appropriate delay and retry count are uncertain, increasing operational complexity.

If the service call is a local call within the same thread accessing the same database, a transaction can guarantee consistency. For distributed calls, distributed transaction mechanisms such as 2PC, 3PC, TCC, or Saga can be used, with middleware like Seata.

The focus, however, is not on distributed transaction solutions but on quick data consistency handling in timeout scenarios.

Thoughts

Timeouts may stem from network jitter, server overload, or poor performance. The goal is to ensure eventual data consistency across applications.

If a service can be degraded, degradation is a fast recovery method; otherwise, retry and compensation are needed.

RPC retries require idempotent interfaces to avoid side effects. Idempotency means multiple identical requests produce the same result as a single request. When idempotency fails (e.g., UUID changes on each retry), fast data recovery becomes challenging.

In such cases, manual intervention to reconcile data is error‑prone and time‑consuming.

Approach

The tool, called a “magic tool,” provides a lightweight mock/spy capability. It can be configured to mock specific method calls and return predefined results, effectively “pre‑paying” success to the caller.

Tool Expectations

Define a configurable range of requests that can be mocked, with dynamic second‑level activation.

For requests within this range, perform pseudo‑idempotent mock and return a specific result to the caller.

The range should be precise to avoid unintended mocking that could cause data inconsistency.

The configuration is expressed as JSON, for example:

{
  "detailList": [
    {
      "enabled": true,
      "className": "com.jdwl.wms.stock.app.service.main.StockTransferAppServiceImpl",
      "methodName": "increaseStock",
      "basicNo1": null,
      "basicNo2": null,
      "basicNo3": "6_6_601",
      "uuidList": null,
      "businessNoList": ["GZQ202503160250001"],
      "startTime": "2025-03-16 01:50:00",
      "endTime": "2025-03-18 03:50:00",
      "strategy": "DO_AND_RETURN_SUCCESS_REGARDLESS_OF_FAILURE",
      "defaultResult": {
        "resultValue": true,
        "resultCode": 100000,
        "prompType": 0,
        "success": true
      }
    }
  ]
}

The strategy can be either DO_NOTHING_AND_RETURN_SPECIFIED_VALUE (mock without executing) or DO_AND_RETURN_SUCCESS_REGARDLESS_OF_FAILURE (execute then return success).

Core Implementation

Matching Configured Range

Mock or Spy by Strategy

Use Cases

Case 1: Mock Service Call

Configure DUCC to define the mock range, then the method is not executed; the predefined result is returned directly.

{
  "detailList": [
    {
      "enabled": true,
      "className": "com.jdwl.wms.stock.app.service.main.StockTransferAppServiceImpl",
      "methodName": "increaseStock",
      "basicNo1": null,
      "basicNo2": null,
      "basicNo3": "6_6_601",
      "uuidList": null,
      "businessNoList": ["GZQ202503160250001"],
      "startTime": "2025-03-16 01:50:00",
      "endTime": "2025-03-18 03:50:00",
      "strategy": "DO_NOTHING_AND_RETURN_SPECIFIED_VALUE",
      "defaultResult": {
        "resultValue": true,
        "resultCode": 100000,
        "prompType": 0,
        "success": true
      }
    }
  ]
}

The database shows that the method was not actually executed, and the mock result was returned.

Case 2: Block Abnormal Data Generation

When abnormal data (e.g., erroneous inventory pre‑allocation) is detected, a configuration can block further generation of such data.

After verification, the tool successfully prevented a portion of abnormal data from being created, saving manual reconciliation effort.

Conclusion

The lightweight mock/spy tool can achieve mock or spy effects for non‑idempotent or idempotency‑failed scenarios, supporting quick data recovery for both write and read services. Although the current version is simple and limited, it demonstrates a practical approach to handling data consistency in timeout situations.

How to Integrate

1. Add Maven dependencies (example shown). 2. Annotate target methods with @Magic (supports SpEL expressions). 3. Configure the mock range via DUCC or Spring YAML, and remove the configuration after use.

<dependency>
  <groupId>com.jd.sword</groupId>
  <artifactId>sword-aspect</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <exclusions> ... </exclusions>
</dependency>
<dependency>
  <groupId>com.jd.sword</groupId>
  <artifactId>sword-constant</artifactId>
  <version>1.0.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>com.jd.sword</groupId>
  <artifactId>sword-annotation</artifactId>
  <version>1.0.0-SNAPSHOT</version>
</dependency>

@Magic(enabled = true, basicNo3 = "#args[0].requestHeader.warehouseNo", uuid = "#args[0].requestHeader.uuid", businessNo = "#args[0].requestHeader.businessNo")

magic:
  content: '{"detailList":[{"enabled":true,"className":"com.jdwl.wms.stock.app.service.main.StockTransferAppServiceImpl","methodName":"increaseStock","basicNo1":null,"basicNo2":null,"basicNo3":"6_6_601","uuidList":null,"businessNoList":["GZQ202503160250"],"startTime":"2025-03-16 01:50:00","endTime":"2025-03-18 03:50:00","strategy":"DO_AND_RETURN_SUCCESS_REGARDLESS_OF_FAILURE","defaultResult":{"resultValue":true,"resultCode":100000,"prompType":0,"success":true}}]}'