How to Prevent Duplicate Requests in Java Using Redis and MD5 Deduplication

In many cases user requests may be sent repeatedly, which is harmless for read‑only queries but can cause severe issues for write operations such as duplicate orders.

Duplicate request scenarios

Replay attacks intercepted by a hacker.

Client or front‑end resends due to network issues or rapid clicks.

Gateway retransmission.

Other cases.

The article focuses on server‑side handling; client‑side click‑prevention is out of scope.

Deduplication using a unique request ID

If each request carries a unique identifier, Redis can be used to store the ID for a short period. When the ID already exists, the request is considered a duplicate.

Sample code

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute(
    (RedisCallback<Boolean>) connection ->
        connection.set(KEY.getBytes(), val.getBytes(),
            Expiration.milliseconds(expireTime),
            RedisStringCommands.SetOption.SET_IF_ABSENT));
boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first access
} else {
    isConsiderDup = true; // duplicate
}

Business‑parameter deduplication

When a unique request ID is unavailable, the request parameters can be used as an identifier. For a single field reqParam, a key like userId:method:reqParam can detect duplicates.

Parameter MD5 digest

For JSON parameters, sort keys, concatenate them, and compute an MD5 hash to obtain a compact identifier.

Code example

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Note: MD5 collisions are theoretically possible but unlikely within a short time window.

Further optimization: exclude time‑dependent fields

Requests that include timestamps or GPS coordinates may differ in those fields while being otherwise identical. Excluding such fields before computing the MD5 ensures that rapid repeated submissions are still recognized as duplicates.

Java utility class for request deduplication

public class ReqDedupHelper {
    /**
     * @param reqJSON request parameters (usually JSON)
     * @param excludeKeys fields to exclude before computing MD5
     * @return MD5 digest of the remaining parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decryptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decryptParam, TreeMap.class);
        if (excludeKeys != null) {
            List<String> dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            return DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
            return null;
        }
    }
}

Test logs

public static void main(String[] args) {
    // two requests differ only by requestTime
    String req = "{
\"requestTime\":\"20190101120001\",
\"requestValue\":\"1000\",
\"requestKey\":\"key\"
}";
    String req2 = "{
\"requestTime\":\"20190101120002\",
\"requestValue\":\"1000\",
\"requestKey\":\"key\"
}";
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + " , req2MD5=" + dedupMD52);
    // exclude requestTime
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + " , req2MD5=" + dedupMD54);
}

Log output shows that without excluding requestTime the MD5 values differ, while after exclusion they match, confirming the deduplication logic.

Complete solution

Combine the parameter MD5 digest (with excluded fields) with a Redis key of the form dedup:U=<userId>M=<method>P=<md5>, set it with a short expiration (e.g., 1000 ms) using an atomic SETNX+expire operation. This provides an efficient, idempotent request‑deduplication mechanism for backend services.