Mastering Idempotent Payment APIs: From Pitfalls to Distributed‑Lock Solutions

Introduction

Little Ming from the neighboring team repeatedly gets rebuked by senior architect Lao Li for neglecting idempotency in a payment interface, prompting a step‑by‑step deep dive into the essence of idempotent design.

What Is Idempotency?

Idempotency : In mathematics and computer science, an idempotent operation yields the same effect no matter how many times it is executed.

First Attempt – Neglected Idempotency

The initial payment controller processes the request directly without any duplicate‑submission protection, leading to multiple charges if a user clicks the pay button repeatedly.

@RestController
public class PaymentController {
    @Autowired
    private PaymentService paymentService;

    @PostMapping("/pay")
    public ResponseEntity<String> pay(@RequestBody PaymentRequest request) {
        // Direct processing, no duplicate‑submission handling
        PaymentResult result = paymentService.processPayment(request);
        return ResponseEntity.ok("Payment succeeded, order ID: " + result.getOrderId());
    }
}

@Service
public class PaymentService {
    public PaymentResult processPayment(PaymentRequest request) {
        // Dangerous: directly deduct amount without any duplicate‑submission guard
        Account account = accountRepository.findById(request.getUserId());
        account.setBalance(account.getBalance().subtract(request.getAmount()));
        accountRepository.save(account);
        // Create payment record
        Payment payment = new Payment();
        payment.setUserId(request.getUserId());
        payment.setAmount(request.getAmount());
        payment.setStatus("SUCCESS");
        return paymentRepository.save(payment);
    }
}

Lao Li points out that network glitches could cause three identical charges.

Second Attempt – Simple Redis Deduplication

Ming tries a quick fix by using Redis with a key composed of user ID and amount to reject duplicate requests.

@RestController
public class PaymentController {
    @Autowired
    private RedisTemplate<String, String> redisTemplate;

    @PostMapping("/pay")
    public ResponseEntity<String> pay(@RequestBody PaymentRequest request) {
        String key = "payment:" + request.getUserId() + ":" + request.getAmount();
        if (redisTemplate.hasKey(key)) {
            return ResponseEntity.ok("Duplicate request, payment already processed");
        }
        redisTemplate.opsForValue().set(key, "processing", 60, TimeUnit.SECONDS);
        PaymentResult result = paymentService.processPayment(request);
        return ResponseEntity.ok("Payment succeeded, order ID: " + result.getOrderId());
    }
}

Lao Li warns that the key design is flawed—different payments with the same amount share the same key, and a failed request leaves a stale key that blocks future payments.

Third Attempt – Idempotent Token

Ming introduces an idempotent token supplied by the client, storing the result in Redis.

@RestController
public class PaymentController {
    @PostMapping("/pay")
    public ResponseEntity<String> pay(@RequestBody PaymentRequest request) {
        String token = request.getIdempotentToken();
        if (StringUtils.isEmpty(token)) {
            return ResponseEntity.badRequest().body("Missing idempotent token");
        }
        String key = "payment:token:" + token;
        if (redisTemplate.hasKey(key)) {
            String result = redisTemplate.opsForValue().get(key);
            return ResponseEntity.ok("Duplicate request: " + result);
        }
        try {
            PaymentResult result = paymentService.processPayment(request);
            redisTemplate.opsForValue().set(key, "SUCCESS:" + result.getOrderId(), 24, TimeUnit.HOURS);
            return ResponseEntity.ok("Payment succeeded, order ID: " + result.getOrderId());
        } catch (Exception e) {
            redisTemplate.opsForValue().set(key, "FAILED:" + e.getMessage(), 24, TimeUnit.HOURS);
            throw e;
        }
    }
}

Although better, Lao Li notes a concurrency flaw: two simultaneous requests can both see the token as unused and process the payment twice.

Correct Idempotent Design

The final solution combines result caching, a double‑checked lock, and state management using Redisson distributed locks.

@RestController
public class PaymentController {
    @Autowired
    private RedissonClient redissonClient;
    @Autowired
    private PaymentService paymentService;

    @PostMapping("/pay")
    public ResponseEntity<PaymentResponse> pay(@RequestBody PaymentRequest request) {
        String idempotentKey = request.getIdempotentToken();
        validateRequest(request, idempotentKey);
        // 1. Quick result lookup
        PaymentResponse existing = paymentService.getPaymentResult(idempotentKey);
        if (existing != null) {
            return ResponseEntity.ok(existing);
        }
        // 2. Distributed lock
        RLock lock = redissonClient.getLock("payment:lock:" + idempotentKey);
        try {
            if (lock.tryLock(3, 10, TimeUnit.SECONDS)) {
                // 3. Double‑check after acquiring lock
                existing = paymentService.getPaymentResult(idempotentKey);
                if (existing != null) {
                    return ResponseEntity.ok(existing);
                }
                PaymentResponse result = paymentService.processPaymentWithIdempotent(request);
                return ResponseEntity.ok(result);
            } else {
                throw new RuntimeException("System busy, please retry later");
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException("Request interrupted");
        } finally {
            if (lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }

    private void validateRequest(PaymentRequest request, String idempotentKey) {
        if (StringUtils.isEmpty(idempotentKey)) {
            throw new IllegalArgumentException("Missing idempotent token");
        }
        // additional business validation …
    }
}

Key points:

Result lookup before acquiring the lock saves time for most duplicate requests.

“Double‑checked locking” prevents race conditions while the lock is being obtained.

Lock wait time is limited to 3 seconds; lock hold time is 10 seconds, balancing safety and latency.

Separate handling of business vs. system exceptions ensures the lock is always released.

Core Business Logic

@Service
@Transactional
public class PaymentService {
    @Autowired
    private PaymentRecordRepository paymentRecordRepository;
    @Autowired
    private AccountRepository accountRepository;

    public PaymentResponse processPaymentWithIdempotent(PaymentRequest request) {
        String idempotentKey = request.getIdempotentToken();
        try {
            PaymentRecord record = createPaymentRecord(request, idempotentKey);
            processPayment(request); // actual payment logic
            record.setStatus(PaymentStatus.SUCCESS);
            record.setCompletedAt(LocalDateTime.now());
            paymentRecordRepository.save(record);
            return PaymentResponse.success(record.getOrderId());
        } catch (Exception e) {
            updatePaymentRecordOnFailure(idempotentKey, e.getMessage());
            throw e;
        }
    }

    private PaymentRecord createPaymentRecord(PaymentRequest request, String idempotentKey) {
        PaymentRecord record = new PaymentRecord();
        record.setIdempotentKey(idempotentKey);
        record.setUserId(request.getUserId());
        record.setAmount(request.getAmount());
        record.setStatus(PaymentStatus.PROCESSING);
        record.setCreatedAt(LocalDateTime.now());
        try {
            return paymentRecordRepository.save(record);
        } catch (DataIntegrityViolationException e) {
            throw new RuntimeException("Duplicate payment request");
        }
    }

    private void updatePaymentRecordOnFailure(String idempotentKey, String errorMsg) {
        PaymentRecord record = paymentRecordRepository.findByIdempotentKey(idempotentKey);
        if (record != null) {
            record.setStatus(PaymentStatus.FAILED);
            record.setErrorMessage(errorMsg);
            record.setCompletedAt(LocalDateTime.now());
            paymentRecordRepository.save(record);
        }
    }

    public PaymentResponse getPaymentResult(String idempotentKey) {
        PaymentRecord record = paymentRecordRepository.findByIdempotentKey(idempotentKey);
        if (record == null) return null;
        switch (record.getStatus()) {
            case SUCCESS:
                return PaymentResponse.success(record.getOrderId());
            case PROCESSING:
                return PaymentResponse.processing();
            case FAILED:
                return null; // allow retry
            default:
                return null;
        }
    }
}

Database unique constraints act as a safety net in case the distributed lock fails.

Different Scenarios and Strategies

Not every system uses Redis, and not all require high‑throughput handling. The article outlines alternatives such as message‑queue‑based async idempotency, sharding by user or region, CRDT‑based eventual consistency, and hybrid sync‑async architectures.

Monitoring & Alerting

Effective idempotent design must be coupled with operational monitoring: thread‑pool metrics, Redis health, and real‑time alerts via DingTalk, Feishu, or email for abnormal business behavior.

Conclusion

A solid idempotent solution prevents duplicate execution while preserving user experience. It balances simplicity and complexity, using the right tools for the traffic level, and ultimately keeps the system’s state consistent even under uncertainty.