Server‑Side Request Deduplication Using Redis and Java

This technical note discusses the problem of duplicate user requests—especially write operations that can cause serious issues such as double orders—and presents a systematic server‑side solution.

1. Unique request ID deduplication – If each request carries a unique identifier, a Redis key can be used to store the ID for a short TTL (e.g., 1000 ms). Subsequent requests with the same ID are considered duplicates.

String KEY = "REQ12343456788"; // request unique ID long expireTime = 1000; // 1000 ms expiration long expireAt = System.currentTimeMillis() + expireTime; String val = "expireAt@" + expireAt; Boolean firstSet = stringRedisTemplate.execute((RedisCallback ) connection -> connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT)); boolean isConsiderDup = (firstSet != null && firstSet) ? false : true;

2. Business‑parameter deduplication – When a unique request ID is not available, the request parameters themselves can be used as a fingerprint. The parameters are sorted, concatenated, and an MD5 hash is computed. Volatile fields such as timestamps or GPS coordinates are excluded before hashing.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

3. Helper class for MD5 fingerprint

public class ReqDedupHelper { /** * @param reqJSON request JSON string * @param excludeKeys fields to remove before hashing * @return MD5 of the remaining parameters */ public String dedupParamMD5(final String reqJSON, String... excludeKeys) { TreeMap paramTreeMap = JSON.parseObject(reqJSON, TreeMap.class); if (excludeKeys != null) { List exclude = Arrays.asList(excludeKeys); for (String key : exclude) { paramTreeMap.remove(key); } } String json = JSON.toJSONString(paramTreeMap); return jdkMD5(json); } private static String jdkMD5(String src) { try { MessageDigest md = MessageDigest.getInstance("MD5"); byte[] bytes = md.digest(src.getBytes()); return DatatypeConverter.printHexBinary(bytes); } catch (Exception e) { log.error("", e); return null; } } }

4. Example usage and test logs

public static void main(String[] args) { String req = "{\n\"requestTime\":\"20190101120001\",\n\"requestValue\":\"1000\",\n\"requestKey\":\"key\"\n}"; String req2 = "{\n\"requestTime\":\"20190101120002\",\n\"requestValue\":\"1000\",\n\"requestKey\":\"key\"\n}"; String md5a = new ReqDedupHelper().dedupParamMD5(req); String md5b = new ReqDedupHelper().dedupParamMD5(req2); System.out.println("req1MD5 = " + md5a + ", req2MD5 = " + md5b); String md5c = new ReqDedupHelper().dedupParamMD5(req, "requestTime"); String md5d = new ReqDedupHelper().dedupParamMD5(req2, "requestTime"); System.out.println("req1MD5 = " + md5c + ", req2MD5 = " + md5d); }

The logs show that without excluding the timestamp the MD5 values differ, while after removing requestTime the hashes match, confirming the deduplication logic.

5. Complete solution – Combine user ID, method name, and the MD5 fingerprint to form a Redis key, set it atomically with a short TTL, and treat a failed SET_IF_ABSENT as a duplicate request.

String userId = "12345678"; String method = "pay"; String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime"); String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5; long expireTime = 1000; long expireAt = System.currentTimeMillis() + expireTime; String val = "expireAt@" + expireAt; Boolean firstSet = stringRedisTemplate.execute((RedisCallback ) conn -> conn.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT)); boolean isDuplicate = !(firstSet != null && firstSet);

Finally, the article adds a brief promotional note about interview questions and community links, but the core technical content provides a reusable pattern for idempotent request handling in backend services.