How to Prevent Duplicate Requests with Redis: A Complete Backend Deduplication Guide

Some users may send duplicate requests in certain situations; for read‑only queries this is usually harmless, but for write operations it can cause severe consequences such as duplicate orders.

Typical duplicate scenarios

Hackers intercept and replay requests.

The front‑end/client resends a request due to network issues or rapid user clicks.

Gateway retransmission.

…

This article focuses on how to handle such cases gracefully on the server side; client‑side measures like disabling rapid clicks are out of scope.

Deduplication using a unique request ID

If each request carries a unique identifier, Redis can be used for deduplication: if the identifier exists in Redis, the request is considered a duplicate.

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration, requests within this window are treated as duplicates
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

// If the Redis key already exists, treat as duplicate
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));

final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first access
} else {
    isConsiderDup = true; // key exists, duplicate
}

Business‑parameter deduplication

When a unique request ID is not available, the request parameters themselves can be used as an identifier. For a simple case with a single parameter reqParam:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

This key uniquely identifies a request from a specific user to a specific interface with the same parameter.

1. Compute a digest of request parameters

For JSON parameters, sort the keys, concatenate them into a string, and compute an MD5 hash to serve as a compact identifier.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Note: MD5 collisions are theoretically possible, but within a short time window (e.g., one second) the risk is negligible.

2. Further optimization: exclude time‑related fields

Requests may contain timestamps or GPS coordinates that differ slightly between repeats. Excluding such fields before hashing prevents false negatives.

// Two requests differ only by requestTime
String req = "{
" +
    "\"requestTime\" :\"20190101120001\",
" +
    "\"requestValue\" :\"1000\",
" +
    "\"requestKey\" :\"key\"
}";

String req2 = "{
" +
    "\"requestTime\" :\"20190101120002\",
" +
    "\"requestValue\" :\"1000\",
" +
    "\"requestKey\" :\"key\"
}";

Deduplication utility class

public class ReqDedupHelper {
    /**
     * @param reqJSON request parameters (usually JSON)
     * @param excludeKeys fields to remove before computing the digest
     * @return MD5 digest of the remaining parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decryptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decryptParam, TreeMap.class);
        if (excludeKeys != null) {
            List<String> dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        String res = null;
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            res = DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
        }
        return res;
    }
}

Test logs

public static void main(String[] args) {
    // Two requests differ only by requestTime
    String req = "{
" +
        "\"requestTime\" :\"20190101120001\",
" +
        "\"requestValue\" :\"1000\",
" +
        "\"requestKey\" :\"key\"
}";

    String req2 = "{
" +
        "\"requestTime\" :\"20190101120002\",
" +
        "\"requestValue\" :\"1000\",
" +
        "\"requestKey\" :\"key\"
}";

    // Full parameter comparison, MD5 differs
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + " , req2MD5=" + dedupMD52);

    // Excluding requestTime, MD5 becomes identical
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + " , req2MD5=" + dedupMD54);
}

Log output:

req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267 , req2MD5 = A2D20BAC78551C4CA09BEF97FE468A3F
req1MD5 = C2A36FED15128E9E878583CAAAFEFDE9 , req2MD5 = C2A36FED15128E9E878583CAAAFEFDE9

Explanation:

Initially the two requests have different requestTime, so their MD5 digests differ.

When requestTime is excluded, the digests match, confirming the deduplication logic works as intended.

Complete solution

Combine the unique request ID (or parameter MD5) with a Redis key, set it with SETNX and an expiration time (e.g., 1000 ms) to atomically detect and block duplicate requests within the defined window.