Server‑Side Request Deduplication Using Unique IDs, Parameter Hashing, and Redis in Java

When user requests are sent repeatedly, especially write operations, they can cause serious problems such as duplicate orders. The article focuses on handling such duplicates on the server side, not on client‑side click‑prevention.

Typical duplicate scenarios include request replay attacks, client‑side repeated submissions, gateway retransmissions, and other cases where the same logical request is sent multiple times.

Deduplication by Unique Request ID – If each request carries a globally unique identifier, Redis can be used to store the ID as a key. As long as the key exists, the request is considered a duplicate. The following Java snippet shows a basic Redis SETNX implementation with a 1000 ms expiration:

String KEY = "REQ12343456788"; // request unique ID long expireTime = 1000; // 1000 ms expiration long expireAt = System.currentTimeMillis() + expireTime; String val = "expireAt@" + expireAt; Boolean firstSet = stringRedisTemplate.execute((RedisCallback ) connection -> connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT)); boolean isConsiderDup = (firstSet != null && firstSet) ? false : true;

Deduplication by Business Parameters – When a unique request ID is not available, a composite key can be built from user ID, method name, and request parameters. For simple single‑field parameters the key looks like:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

For JSON parameters, the article proposes sorting the JSON keys, concatenating them, and computing an MD5 hash to keep the key short. The MD5 digest serves as the parameter identifier:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Optimizing by Removing Time‑Sensitive Fields – Repeated requests often differ only in timestamp or GPS fields, causing different keys even though the logical operation is the same. By excluding such fields before hashing, the deduplication becomes more reliable.

Utility Class ReqDedupHelper – The article provides a reusable Java class that parses a JSON request into a TreeMap , removes configurable exclude keys, converts the map back to JSON, and returns the MD5 digest. It also contains a private method for computing the MD5 using MessageDigest :

public class ReqDedupHelper { public String dedupParamMD5(final String reqJSON, String... excludeKeys) { TreeMap paramTreeMap = JSON.parseObject(reqJSON, TreeMap.class); if (excludeKeys != null) { List dedupExcludeKeys = Arrays.asList(excludeKeys); if (!dedupExcludeKeys.isEmpty()) { for (String dedupExcludeKey : dedupExcludeKeys) { paramTreeMap.remove(dedupExcludeKey); } } } String paramTreeMapJSON = JSON.toJSONString(paramTreeMap); return jdkMD5(paramTreeMapJSON); } private static String jdkMD5(String src) { try { MessageDigest messageDigest = MessageDigest.getInstance("MD5"); byte[] mdBytes = messageDigest.digest(src.getBytes()); return DatatypeConverter.printHexBinary(mdBytes); } catch (Exception e) { log.error("", e); return null; } } }

Test code demonstrates that two JSON strings differing only by requestTime produce different MD5 values, but after excluding requestTime the MD5s match, confirming the approach.

public static void main(String[] args) { String req = "{\n\"requestTime\":\"20190101120001\",\n\"requestValue\":\"1000\",\n\"requestKey\":\"key\"\n}"; String req2 = "{\n\"requestTime\":\"20190101120002\",\n\"requestValue\":\"1000\",\n\"requestKey\":\"key\"\n}"; String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req); String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2); System.out.println("req1MD5 = " + dedupMD5 + ", req2MD5=" + dedupMD52); String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime"); String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime"); System.out.println("req1MD5 = " + dedupMD53 + ", req2MD5=" + dedupMD54); }

The console output shows different MD5s for the full payloads and identical MD5s when requestTime is excluded.

Complete Solution – Combining the parameter‑hashing utility with the Redis atomic SETNX logic yields a full deduplication workflow:

String userId = "12345678"; // user String method = "pay"; // API name String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime"); String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5; long expireTime = 1000; // 1 second window long expireAt = System.currentTimeMillis() + expireTime; String val = "expireAt@" + expireAt; Boolean firstSet = stringRedisTemplate.execute((RedisCallback ) connection -> connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT)); boolean isConsiderDup = (firstSet != null && firstSet) ? false : true;

This approach ensures that duplicate write requests within a short time window are detected and blocked, while allowing legitimate distinct requests to proceed.