Server‑Side Request Deduplication Using Redis and Java

Duplicate user requests—whether caused by replay attacks, client retries, or gateway resends—can lead to serious problems, especially for write operations; this article focuses on server‑side strategies to detect and reject such duplicates.

Utilizing a Unique Request Identifier

When each request carries a unique ID, Redis can be used to store the ID as a key; if the key already exists, the request is considered a duplicate. The following Java snippet demonstrates the approach with a 1000 ms expiration.

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
boolean isConsiderDup = (firstSet != null && firstSet) ? false : true;

Deduplication Based on Business Parameters

If a request lacks a unique ID, a composite key can be built from user ID, method name, and a hash of the request parameters. For JSON payloads, sorting keys and computing an MD5 digest yields a compact identifier.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

When request parameters contain volatile fields such as timestamps or GPS coordinates, those fields should be excluded before computing the MD5 to avoid false negatives.

Request‑Deduplication Helper Class (Java)

public class ReqDedupHelper {
    /**
     * @param reqJSON   request JSON string
     * @param excludeKeys fields to remove before hashing
     * @return MD5 digest of the cleaned parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decrptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decrptParam, TreeMap.class);
        if (excludeKeys != null) {
            List
dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            return DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
            return null;
        }
    }
}

Test Logs

public static void main(String[] args) {
    // two requests differ only by requestTime
    String req = "{\n" +
        "\"requestTime\" :\"20190101120001\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n}";
    String req2 = "{\n" +
        "\"requestTime\" :\"20190101120002\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n}";
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + " , req2MD5=" + dedupMD52);
    // remove requestTime before hashing
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + " , req2MD5=" + dedupMD54);
}

Output shows different MD5 values when the timestamp is included and identical values when it is excluded, confirming the effectiveness of the deduplication helper.

Summary

The complete solution combines a user‑ID/method‑based key, an MD5 digest of cleaned request parameters, a short Redis TTL (e.g., 1000 ms), and an atomic SETNX‑with‑expiration operation to reliably filter duplicate requests within the defined time window.