How to Gracefully Prevent Duplicate Requests on the Server Side

Problem Overview

Duplicate requests, especially write‑type APIs, can cause issues such as double ordering. Common causes include replay attacks, client retries, gateway retransmission, etc. The server should enforce idempotency within a short time window.

Deduplication Using a Unique Request Identifier

If the client supplies a globally unique request ID, the server can store this ID in Redis with a short TTL. Subsequent requests that contain the same ID are treated as duplicates.

String KEY = "REQ12343456788"; // unique request ID
long expireTime = 1000L; // 1000 ms TTL
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

Boolean firstSet = stringRedisTemplate.execute(
    (RedisCallback<Boolean>) connection ->
        connection.set(
            KEY.getBytes(),
            val.getBytes(),
            Expiration.milliseconds(expireTime),
            RedisStringCommands.SetOption.SET_IF_ABSENT));

boolean isDuplicate = !(firstSet != null && firstSet);

Deduplication Based on Business Parameters

When a request does not carry an explicit ID, the request payload itself can be used as a fingerprint. For simple single‑parameter calls a composite key such as userId:method:reqParam is sufficient:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

For JSON payloads the object is first sorted by key, concatenated into a canonical string, and then hashed (MD5). The hash replaces the raw JSON in the Redis key, keeping the key short.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Excluding Time‑Sensitive Fields

Fields that change on each request (e.g., timestamps, GPS coordinates) must be removed before hashing, otherwise identical business requests would be considered different. The helper class below accepts a var‑args list of field names to exclude.

public class ReqDedupHelper {
    /**
     * @param reqJSON   JSON string of the request parameters
     * @param excludeKeys field names to remove before hashing
     * @return MD5 digest of the remaining parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        TreeMap<String, Object> paramTreeMap = JSON.parseObject(reqJSON, TreeMap.class);
        if (excludeKeys != null) {
            for (String key : excludeKeys) {
                paramTreeMap.remove(key);
            }
        }
        String canonicalJson = JSON.toJSONString(paramTreeMap);
        return jdkMD5(canonicalJson);
    }

    private static String jdkMD5(String src) {
        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            byte[] bytes = md.digest(src.getBytes());
            return DatatypeConverter.printHexBinary(bytes);
        } catch (Exception e) {
            // In production code replace with proper logging
            return null;
        }
    }
}

Verification Example

The following test demonstrates that two JSON payloads differing only by a requestTime field produce different MD5 values, while the same payloads produce identical values when requestTime is excluded.

String req = "{
\"requestTime\":\"20190101120001\",
\"requestValue\":\"1000\",
\"requestKey\":\"key\"
}";
String req2 = "{
\"requestTime\":\"20190101120002\",
\"requestValue\":\"1000\",
\"requestKey\":\"key\"
}";

String md5A = new ReqDedupHelper().dedupParamMD5(req);
String md5B = new ReqDedupHelper().dedupParamMD5(req2);
System.out.println("md5A = " + md5A + ", md5B = " + md5B);

String md5C = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
String md5D = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
System.out.println("md5C = " + md5C + ", md5D = " + md5D);

Without excluding requestTime, the two MD5 hashes differ.

When requestTime is excluded, the hashes are identical, indicating a duplicate request.

Full Idempotency Solution

Combine the parameter‑hashing helper with the Redis SETNX‑plus‑expiration pattern to achieve atomic, time‑bounded deduplication for write‑type APIs.

String userId = "12345678";
String method = "pay";
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;

long expireTime = 1000L; // 1 second window
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

Boolean firstSet = stringRedisTemplate.execute(
    (RedisCallback<Boolean>) connection ->
        connection.set(
            KEY.getBytes(),
            val.getBytes(),
            Expiration.milliseconds(expireTime),
            RedisStringCommands.SetOption.SET_IF_ABSENT));

boolean isDuplicate = !(firstSet != null && firstSet);

This logic guarantees that a write request is processed at most once within the configured window, while read‑only queries can bypass the deduplication step.