Server‑Side Request Deduplication Using Redis and MD5 in Java

When user requests are unintentionally sent multiple times—whether due to replay attacks, client‑side retries, or gateway retransmissions—duplicate processing can cause severe problems, especially for write operations such as order placement.

The article focuses on server‑side strategies for gracefully handling duplicate requests, excluding client‑side click‑prevention from the discussion.

01. Deduplication with a Unique Request Identifier

If each request carries a unique identifier, Redis can be used to store the identifier temporarily; a subsequent request with the same identifier is considered duplicate.

Sample code:

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration, duplicates within this window are rejected
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

// If the Redis key already exists, the request is duplicate
Boolean firstSet = stringRedisTemplate.execute((RedisCallback) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));

final boolean isConsiderDup;
if (firstSet != null && firstSet) { // first access
    isConsiderDup = false;
} else { // key already exists, treat as duplicate
    isConsiderDup = true;
}

02. Business‑Parameter Deduplication

When a unique request ID is not available, the request parameters themselves can be used as a fingerprint. For a simple case with a single parameter reqParam , a composite key can be built from user ID, method name, and the parameter:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

This works when the same user calls the same API with identical parameters.

03. Hashing the Request Parameters

For JSON payloads, sorting the keys alphabetically and computing an MD5 hash yields a compact identifier. The hash replaces the raw JSON in the Redis key:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Although MD5 collisions are theoretically possible, they are negligible within a short deduplication window (e.g., one second).

04. Optimizing by Excluding Time‑Sensitive Fields

Requests that contain timestamps or rapidly changing fields (e.g., GPS coordinates) may generate different hashes even though the logical operation is the same. By stripping such fields before hashing, true duplicates are detected.

Example of two JSON strings that differ only by requestTime :

// Two requests identical except for requestTime
String req = "{\n" +
    "\"requestTime\" :\"20190101120001\",\n" +
    "\"requestValue\" :\"1000\",\n" +
    "\"requestKey\" :\"key\"\n" +
    "}";

String req2 = "{\n" +
    "\"requestTime\" :\"20190101120002\",\n" +
    "\"requestValue\" :\"1000\",\n" +
    "\"requestKey\" :\"key\"\n" +
    "}";

05. Request‑Deduplication Helper Class (Java)

public class ReqDedupHelper {

    /**
     * @param reqJSON   The request payload, usually JSON.
     * @param excludeKeys   Keys to remove before computing the MD5.
     * @return MD5 digest of the filtered parameters.
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decryptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decryptParam, TreeMap.class);
        if (excludeKeys != null) {
            List dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        String res = null;
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            res = DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
        }
        return res;
    }
}

Test program demonstrating the effect of excluding requestTime :

public static void main(String[] args) {
    // Two requests with different requestTime values
    String req = "{\n" +
        "\"requestTime\" :\"20190101120001\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n" +
        "}";
    String req2 = "{\n" +
        "\"requestTime\" :\"20190101120002\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n" +
        "}";
    // Full parameter comparison – MD5 differs
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + " , req2MD5=" + dedupMD52);

    // Excluding requestTime – MD5 becomes identical
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + " , req2MD5=" + dedupMD54);
}

Log output:

req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267 , req2MD5 = A2D20BAC78551C4CA09BEF97FE468A3F
req1MD5 = C2A36FED15128E9E878583CAAAFEFDE9 , req2MD5 = C2A36FED15128E9E878583CAAAFEFDE9

The first line shows different MD5 values because the timestamps differ; the second line shows identical MD5 values after removing requestTime , confirming the deduplication logic.

06. Summary of the Complete Solution

Putting everything together, the final deduplication workflow is:

String userId = "12345678"; // user identifier
String method = "pay"; // API name
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime"); // hash without time field
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;

long expireTime = 1000; // 1 second window
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

// Atomic SETNX + expiration using low‑level Redis API
Boolean firstSet = stringRedisTemplate.execute((RedisCallback) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));

final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // not a duplicate
} else {
    isConsiderDup = true; // duplicate request
}

This approach ensures that identical requests (after optional field exclusion) arriving within the configured time window are recognized and rejected, preventing unintended side effects such as double orders.