Graceful Server‑Side Request Deduplication with Redis and Java

Some user requests may be sent repeatedly; while duplicate queries are harmless, duplicate write operations (e.g., transaction orders) can cause serious issues.

Typical duplication scenarios include:

Hacker intercepts and replays the request.

Frontend/client resends due to network issues or rapid user clicks.

Gateway retransmission.

Other unknown cases.

This article focuses on handling duplication on the server side; client‑side measures such as disabling repeated clicks are out of scope.

Deduplication using a unique request ID

If each request carries a unique identifier, Redis can be used to store the ID; the presence of the key indicates the request has already been processed.

Example code:

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration, duplicates within 1 s are considered repeats
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

// If the Redis key already exists, treat the request as duplicate
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime),
    RedisStringCommands.SetOption.SET_IF_ABSENT));

final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first access
} else {
    isConsiderDup = true; // key exists, duplicate
}

Business‑parameter deduplication

The above works when a unique request ID is available. Often, requests lack such an ID, so we can construct a signature from request parameters, e.g., userId + method + reqParam.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

When the request payload is a JSON object, we can sort its keys, concatenate them, and compute an MD5 hash to use as the signature. To keep the key short, the MD5 digest replaces the full JSON string.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Although MD5 collisions are theoretically possible, they are negligible within a short deduplication window (e.g., one second).

Further optimization: exclude time‑related fields

Requests often contain a timestamp field that varies between otherwise identical calls, causing different MD5 values. By removing such fields before hashing, rapid duplicate clicks can still be detected.

// Two requests differ only by requestTime
String req = "{
" +
    "\"requestTime\" :\"20190101120001\",
" +
    "\"requestValue\" :\"1000\",
" +
    "\"requestKey\" :\"key\"
}";

String req2 = "{
" +
    "\"requestTime\" :\"20190101120002\",
" +
    "\"requestValue\" :\"1000\",
" +
    "\"requestKey\" :\"key\"
}";

Fields like timestamps or GPS coordinates should be excluded before computing the MD5.

Request deduplication utility class (Java)

public class ReqDedupHelper {
    /**
     * @param reqJSON   request parameters, usually JSON
     * @param excludeKeys fields to remove before hashing
     * @return MD5 digest of the remaining parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decrptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decrptParam, TreeMap.class);
        if (excludeKeys != null) {
            List<String> dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        String res = null;
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            res = DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
        }
        return res;
    }
}

Test logs:

public static void main(String[] args) {
    // Two requests differ only by requestTime
    String req = "{
" +
        "\"requestTime\" :\"20190101120001\",
" +
        "\"requestValue\" :\"1000\",
" +
        "\"requestKey\" :\"key\"
}";

    String req2 = "{
" +
        "\"requestTime\" :\"20190101120002\",
" +
        "\"requestValue\" :\"1000\",
" +
        "\"requestKey\" :\"key\"
}";

    // Full comparison, MD5 differs
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + " , req2MD5=" + dedupMD52);

    // Exclude requestTime, MD5 matches
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + " , req2MD5=" + dedupMD54);
}

Log output:

req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267 , req2MD5 = A2D20BAC78551C4CA09BEF97FE468A3F
req1MD5 = C2A36FED15128E9E878583CAAAFEFDE9 , req2MD5 = C2A36FED15128E9E878583CAAAFEFDE9

Explanation:

The first run shows different MD5 values because the timestamps differ.

The second run excludes requestTime, resulting in identical MD5 values, confirming the deduplication works as intended.

Final deduplication solution

String userId = "12345678"; // user
String method = "pay"; // API name
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime"); // compute MD5, excluding time
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;

long expireTime = 1000; // 1000 ms expiration
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

// SETNX with expiration must be atomic; use low‑level API
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime),
    RedisStringCommands.SetOption.SET_IF_ABSENT));

final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false;
} else {
    isConsiderDup = true;
}

Source: 薛定谔的风口猪 – https://jaskey.github.io/blog/2020/05/19/handle-duplicate-request/