Server‑Side Request Deduplication with Unique IDs and Redis in Java

The author discusses the problems caused by duplicate user requests, especially for write operations such as order creation, and lists typical duplication scenarios like replay attacks, client retries, and gateway resends.

To handle deduplication on the server, the article first proposes using a unique request ID stored in Redis; if the key already exists, the request is considered a duplicate.

When a unique ID is not available, the solution extracts a stable identifier from request parameters: it sorts the JSON fields, concatenates them, and computes an MD5 hash. Time‑related fields (e.g., requestTime ) can be excluded to avoid false negatives.

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime),
    RedisStringCommands.SetOption.SET_IF_ABSENT));
boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first access
} else {
    isConsiderDup = true; // duplicate
}

A reusable helper class ReqDedupHelper is provided to compute the MD5 of request JSON after removing specified keys:

public class ReqDedupHelper {
    /**
     * @param reqJSON   request JSON string
     * @param excludeKeys keys to remove before hashing
     * @return MD5 digest of the remaining parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        TreeMap paramTreeMap = JSON.parseObject(reqJSON, TreeMap.class);
        if (excludeKeys != null) {
            List
dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        return jdkMD5(paramTreeMapJSON);
    }
    private static String jdkMD5(String src) {
        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            byte[] mdBytes = md.digest(src.getBytes());
            return DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
            return null;
        }
    }
}

Test code demonstrates that without excluding requestTime the two JSON payloads produce different MD5 values, while excluding the time field makes the digests identical, confirming the deduplication logic.

public static void main(String[] args) {
    String req = "{\n\"requestTime\":\"20190101120001\",\n\"requestValue\":\"1000\",\n\"requestKey\":\"key\"\n}";
    String req2 = "{\n\"requestTime\":\"20190101120002\",\n\"requestValue\":\"1000\",\n\"requestKey\":\"key\"\n}";
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + ", req2MD5=" + dedupMD52);
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + ", req2MD5=" + dedupMD54);
}

The final solution combines the unique‑ID approach and the MD5‑based parameter deduplication, showing the complete Redis key construction, expiration handling, and duplicate detection logic.

In summary, the article provides a practical, end‑to‑end deduplication strategy for backend services using Java and Redis.