How to Prevent Duplicate Requests in Java Services with Redis and MD5

Why Duplicate Requests Matter

Some user requests may be sent repeatedly; while read‑only queries are harmless, write operations can lead to severe problems such as duplicate orders.

Typical Duplication Scenarios

Hacker intercepts and replays the request.

Frontend or client resends due to network issues or rapid clicks.

Gateway retransmission.

Other unknown causes.

The article focuses on server‑side handling; client‑side measures like disabling repeated clicks are out of scope.

Deduplication with a Unique Request ID

If each request carries a unique identifier, Redis can be used to filter duplicates. The key is stored with a short expiration; a second request with the same key is considered a duplicate.

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime),
        RedisStringCommands.SetOption.SET_IF_ABSENT));
boolean isConsiderDup = !(firstSet != null && firstSet);

This uses the atomic SET_IF_ABSENT (SETNX) with expiration to ensure thread‑safe deduplication.

Deduplication Based on Business Parameters

When no unique ID is available, the request parameters themselves can form a fingerprint. A simple composite key can be built from user ID, method name, and a request parameter:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

For JSON payloads, sorting keys and computing an MD5 hash provides a compact identifier. Time‑related fields (e.g., requestTime) should be excluded to avoid false negatives.

public class ReqDedupHelper {
    /**
     * Compute MD5 of request JSON after removing specified keys.
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        TreeMap paramMap = JSON.parseObject(reqJSON, TreeMap.class);
        if (excludeKeys != null) {
            for (String k : excludeKeys) {
                paramMap.remove(k);
            }
        }
        String json = JSON.toJSONString(paramMap);
        return jdkMD5(json);
    }
    private static String jdkMD5(String src) {
        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            byte[] bytes = md.digest(src.getBytes());
            return DatatypeConverter.printHexBinary(bytes);
        } catch (Exception e) {
            return null;
        }
    }
}

Test logs show that without excluding requestTime the MD5 values differ, while excluding it yields identical hashes, confirming the approach.

Putting It All Together

The final solution combines both techniques: generate a deduplication key from user ID, method, and the MD5 of filtered request parameters, then store it in Redis with a short TTL using an atomic SET_IF_ABSENT operation.

String userId = "12345678";
String method = "pay";
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;
long expireTime = 1000;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) conn ->
    conn.set(KEY.getBytes(), ("expireAt@" + (System.currentTimeMillis() + expireTime)).getBytes(),
        Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
boolean isDuplicate = !(firstSet != null && firstSet);

With this approach, duplicate write requests are gracefully filtered on the server side, ensuring idempotent behavior.