Server‑Side Request Deduplication Strategies in Java

Duplicate user requests can cause serious problems, especially for write operations such as order placement; the article discusses server‑side strategies to gracefully handle such cases while ignoring client‑side duplicate‑click prevention.

Typical duplicate scenarios include replay attacks, client retries, gateway retransmissions, and rapid repeated clicks.

Approach 1 – Unique Request ID Deduplication : If each request carries a unique identifier, Redis can be used to store the ID with a short expiration; a subsequent request with the same ID is considered duplicate.

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first access
} else {
    isConsiderDup = true; // duplicate
}

Approach 2 – Business Parameter Key Deduplication : When a unique request ID is not available, combine stable fields (e.g., userId, method name, and a single request parameter) to form a Redis key.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

If the request payload is a JSON object, the key can become excessively long; therefore, the JSON is sorted by key and its MD5 digest is used as the identifier.

Approach 3 – MD5 Digest of Sorted JSON Parameters : The JSON is parsed into a TreeMap (which sorts keys), optional noisy fields such as timestamps are removed, the map is serialized back to JSON, and an MD5 hash is computed.

public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
    String decryptParam = reqJSON;
    TreeMap paramTreeMap = JSON.parseObject(decryptParam, TreeMap.class);
    if (excludeKeys != null) {
        List
dedupExcludeKeys = Arrays.asList(excludeKeys);
        if (!dedupExcludeKeys.isEmpty()) {
            for (String dedupExcludeKey : dedupExcludeKeys) {
                paramTreeMap.remove(dedupExcludeKey);
            }
        }
    }
    String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
    String md5deDupParam = jdkMD5(paramTreeMapJSON);
    log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
    return md5deDupParam;
}

private static String jdkMD5(String src) {
    try {
        MessageDigest messageDigest = MessageDigest.getInstance("MD5");
        byte[] mdBytes = messageDigest.digest(src.getBytes());
        return DatatypeConverter.printHexBinary(mdBytes);
    } catch (Exception e) {
        log.error("", e);
        return null;
    }
}

Testing code demonstrates that two requests differing only by a requestTime field produce different MD5 values, while excluding that field yields identical digests, confirming the deduplication logic.

public static void main(String[] args) {
    String req = "{\n" +
        "\"requestTime\" :\"20190101120001\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n}";
    String req2 = "{\n" +
        "\"requestTime\" :\"20190101120002\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n}";
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + ", req2MD5=" + dedupMD52);
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + ", req2MD5=" + dedupMD54);
}

Log output shows different MD5 values when timestamps are included and identical values when the timestamp is excluded.

req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267 , req2MD5=A2D20BAC78551C4CA09BEF97FE468A3F
req1MD5 = C2A36FED15128E9E878583CAAAFEFDE9 , req2MD5=C2A36FED15128E9E878583CAAAFEFDE9

Complete Solution : Combine the user ID, method name, and the MD5 digest (with time fields excluded) to form the Redis key, set it with a short TTL using an atomic SETNX‑plus‑expire operation, and treat a failed set as a duplicate request.

String userId = "12345678"; // user
String method = "pay"; // API name
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;
long expireTime = 1000; // 1 second window
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false;
} else {
    isConsiderDup = true;
}

Thus, the article provides a full deduplication solution covering unique IDs, business‑parameter keys, MD5 hashing, exclusion of volatile fields, and atomic Redis operations.