Server‑Side Request Deduplication Using Unique IDs and Parameter Hashing in Java

When user requests are repeated, read‑only queries may tolerate duplication, but write operations such as payment can cause serious issues if processed multiple times. Common duplication scenarios include replay attacks, client‑side resubmissions, and gateway retries.

Deduplication with a Unique Request ID

If each request carries a unique identifier, Redis can be used to store the ID temporarily; a subsequent request with the same ID is considered a duplicate.

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first access
} else {
    isConsiderDup = true; // duplicate
}

Business‑Parameter Deduplication

When a unique request ID is not available, the request parameters themselves can be used as a fingerprint. A simple key can be built from user ID, method name, and a parameter value:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

For JSON payloads, the entire object is sorted by key, concatenated, and an MD5 hash is taken to obtain a compact identifier.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Because MD5 collisions are extremely unlikely within a short time window, this approach is safe for typical idempotency checks.

Excluding Volatile Fields (e.g., timestamps)

Requests that contain fields changing on each call, such as a timestamp or GPS coordinates, should have those fields removed before hashing; otherwise identical logical requests would be treated as different.

// Two requests differ only by requestTime
String req = "{\n" +
    "\"requestTime\" :\"20190101120001\",\n" +
    "\"requestValue\" :\"1000\",\n" +
    "\"requestKey\" :\"key\"\n}";
String req2 = "{\n" +
    "\"requestTime\" :\"20190101120002\",\n" +
    "\"requestValue\" :\"1000\",\n" +
    "\"requestKey\" :\"key\"\n}";

Java Deduplication Helper Class

public class ReqDedupHelper {
    /**
     * @param reqJSON   request JSON string
     * @param excludeKeys keys to remove before hashing
     * @return MD5 digest of the filtered parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decrptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decrptParam, TreeMap.class);
        if (excludeKeys != null) {
            List
dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            byte[] mdBytes = md.digest(src.getBytes());
            return DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
            return null;
        }
    }
}

Test Logs

public static void main(String[] args) {
    // two requests differ only by requestTime
    String req = "{\n" +
        "\"requestTime\" :\"20190101120001\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n}";
    String req2 = "{\n" +
        "\"requestTime\" :\"20190101120002\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n}";
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + " , req2MD5=" + dedupMD52);
    // exclude requestTime
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + " , req2MD5=" + dedupMD54);
}

Output shows different MD5 values when the timestamp is included and identical values when it is excluded, confirming the effectiveness of the exclusion logic.

Complete Solution Overview

String userId = "12345678"; // user
String method = "pay"; // API name
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime"); // hash without timestamp
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;
long expireTime = 1000; // 1 s window
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
boolean isConsiderDup = (firstSet != null && firstSet) ? false : true;

This approach guarantees atomic SETNX + expiration using the low‑level Redis API, preventing race conditions where a key could be set without an expiry.

For further reading, the author also links to related articles on Java pooling, permission system design, Nacos internals, and Spring Cloud Gateway OAuth2 integration.