Server‑Side Duplicate Request Handling with Unique IDs and Parameter Deduplication in Java

Why Duplicate Requests Matter

Repeated user requests can be harmless for read operations but may cause severe issues for write operations such as duplicate orders in a transaction interface.

Typical duplicate scenarios include request replay by attackers, client‑side resubmission due to network glitches or rapid user clicks, and gateway retransmission.

This article focuses on server‑side unified handling of such duplicates, not on preventing client‑side repeated clicks.

Deduplication Using a Unique Request ID

If each request carries a unique identifier, Redis can be used to store the ID temporarily; the presence of the ID indicates the request has already been processed.

Sample code:

String KEY = "REQ12343456788"; // request unique ID
    long expireTime = 1000; // 1000 ms expiration, duplicates within this window are considered repeats
    long expireAt = System.currentTimeMillis() + expireTime;
    String val = "expireAt@" + expireAt;

    // If the Redis key already exists, treat the request as duplicate
    Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
        connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));

    final boolean isConsiderDup;
    if (firstSet != null && firstSet) {
        isConsiderDup = false; // first access
    } else {
        isConsiderDup = true; // key exists, duplicate
    }

Business Parameter Deduplication

When a unique request ID is unavailable, the request parameters themselves can serve as an identifier. A simple approach concatenates user ID, method name, and a single parameter value (e.g., userId:method:reqParam ).

For complex JSON payloads, sort the keys alphabetically, concatenate them, and compute an MD5 hash to obtain a compact identifier.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

Because some fields (e.g., timestamps or GPS coordinates) vary between otherwise identical requests, they should be excluded before hashing.

Excluding Time Fields

Example requests differing only by a requestTime field:

// Two requests with a one‑second difference in requestTime
String req = "{\n" +
    "\"requestTime\" :\"20190101120001\",\n" +
    "\"requestValue\" :\"1000\",\n" +
    "\"requestKey\" :\"key\"\n}";

String req2 = "{\n" +
    "\"requestTime\" :\"20190101120002\",\n" +
    "\"requestValue\" :\"1000\",\n" +
    "\"requestKey\" :\"key\"\n}";

By removing requestTime before hashing, the two requests produce the same MD5, allowing them to be treated as duplicates.

Java Deduplication Helper Class

public class ReqDedupHelper {
    /**
     * @param reqJSON   the request parameters, usually a JSON string
     * @param excludeKeys   fields to remove before computing the digest
     * @return MD5 digest of the filtered parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decrptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decrptParam, TreeMap.class);
        if (excludeKeys != null) {
            List
dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        String res = null;
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            res = DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
        }
        return res;
    }
}

Test logs demonstrate that when requestTime is excluded, both requests yield the same MD5 value.

public static void main(String[] args) {
    // Two requests with a one‑second difference in requestTime
    String req = "{\n" +
        "\"requestTime\" :\"20190101120001\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n}";

    String req2 = "{\n" +
        "\"requestTime\" :\"20190101120002\",\n" +
        "\"requestValue\" :\"1000\",\n" +
        "\"requestKey\" :\"key\"\n}";

    // Full comparison – MD5 differs
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = " + dedupMD5 + " , req2MD5=" + dedupMD52);

    // Excluding requestTime – MD5 identical
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2, "requestTime");
    System.out.println("req1MD5 = " + dedupMD53 + " , req2MD5=" + dedupMD54);
}

Log output:

req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267 , req2MD5 = A2D20BAC78551C4CA09BEF97FE468A3F
req1MD5 = C2A36FED15128E9E878583CAAAFEFDE9 , req2MD5 = C2A36FED15128E9E878583CAAAFEFDE9

Explanation: The first pair differs because of different timestamps; the second pair matches after excluding the timestamp, confirming the deduplication logic.

Complete Solution

Putting everything together, the final deduplication workflow is:

String userId = "12345678"; // user identifier
String method = "pay"; // API name
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime"); // compute MD5 without timestamp
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;

long expireTime = 1000; // 1 second window
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

// Atomic SETNX with expiration using low‑level Redis API
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));

final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first request, process normally
} else {
    isConsiderDup = true; // duplicate within the window
}