How to Gracefully Prevent Duplicate Requests with Redis and Java

Duplicate requests can occur in various scenarios and may cause serious problems for write operations such as order placement.

Typical duplicate scenarios include hacker replay attacks, client resends due to network issues or rapid user clicks, and gateway retransmissions.

The article focuses on server‑side handling of duplicate requests.

Deduplication Using Unique Request IDs

If each request carries a unique identifier, Redis can be used to reject repeats by checking whether the ID already exists.

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration, repeats within this window are considered duplicates
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first access
} else {
    isConsiderDup = true; // key already exists, treat as duplicate
}

Business Parameter Deduplication

When no unique ID is available, combine user ID, method name, and request parameters to form a deduplication key.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

Hashing Request Parameters

For JSON payloads, sort the keys, concatenate them, and compute an MD5 hash to use as the parameter identifier.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Excluding Time‑Sensitive Fields

Rapid duplicate clicks may include a timestamp field that changes each request; exclude such fields (e.g., requestTime) before hashing.

// Two requests differ only by requestTime
String req = "{
" +
    "\"requestTime\" :\"20190101120001\",
" +
    "\"requestValue\" :\"1000\",
" +
    "\"requestKey\" :\"key\"
}";
String req2 = "{
" +
    "\"requestTime\" :\"20190101120002\",
" +
    "\"requestValue\" :\"1000\",
" +
    "\"requestKey\" :\"key\"
}";

Request Deduplication Utility Class (Java)

public class ReqDedupHelper {
    /**
     * @param reqJSON   request parameters, usually JSON
     * @param excludeKeys fields to remove before computing the MD5
     * @return MD5 digest of the filtered parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decrptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decrptParam, TreeMap.class);
        if (excludeKeys != null) {
            List<String> dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        String res = null;
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            res = DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
        }
        return res;
    }
}

Test logs demonstrate that without excluding requestTime the MD5 values differ, while after exclusion they match, confirming the deduplication logic.

Summary

The complete deduplication solution combines a user‑ID/method‑based key with an MD5 of the request parameters (excluding volatile fields), stores the key in Redis with a short TTL using an atomic SETNX‑plus‑expire operation, and checks the result to decide whether the request is a duplicate.