Ensuring Idempotency and Preventing Double Payments in a Distributed Payments System

In a micro‑service (SOA) environment, payment systems must balance performance and consistency, which introduces design complexity. Airbnb’s global payment platform faces this challenge and adopts distributed‑transaction techniques to maintain both.

What is Idempotency?

Idempotent API calls allow clients to repeat the same request without changing the result, ensuring that a payment is processed exactly once even if the request is retried.

Idempotency is crucial for financial operations where duplicate processing must be avoided.

Problem Description

The system must provide eventual consistency for payments, handle service timeouts, lost responses, and duplicate user actions, while keeping latency low and avoiding custom per‑use‑case solutions.

Solution

Airbnb built a generic, configurable idempotency library named Orpheus . It isolates idempotency logic from business code, uses a single idempotency key per request, and classifies errors as retryable or non‑retryable.

Idempotency key is passed to the framework to identify each request.

All idempotency data is stored in the primary database to guarantee strong consistency.

Java lambdas combine multiple database commits into a single transaction.

Errors are categorized as “retryable” or “non‑retryable”.

Minimizing Database Commits

Orpheus splits an API call into three phases: Pre‑RPC, RPC, and Post‑RPC. Database writes occur only in Pre‑RPC and Post‑RPC, while the RPC phase performs network calls without touching the database, preserving atomicity.

Pre‑RPC: Record request details in the database.

RPC: Perform external service calls (e.g., payment gateways).

Post‑RPC: Record response details and error classification.

Java Lambdas Combining Transactions

Java lambda expressions enable seamless composition of multiple commits into a single DB transaction. A simplified example of using Orpheus is shown below:

public Response processPayment(InitiatePaymentRequest request, UriInfo uriInfo)
   throws YourCustomException {
   return orpheusManager.process(
     request.getIdempotencyKey(),
     uriInfo,
     // 1. Pre‑RPC
     () -> {
       PaymentRequestResource paymentRequestResource = recordPaymentRequest(request);
       return Optional.of(paymentRequestResource);
     },
     // 2. RPC
     (isRetry, paymentRequest) -> {
       return executePayment(paymentRequest, isRetry);
     },
     // 3. Post‑RPC
     (isRetry, paymentResponse) -> {
       return recordPaymentResponse(paymentResponse);
     });
}

A more detailed implementation demonstrates how Orpheus creates or finds the idempotency request, executes the three phases, and handles exceptions:

public
Response process(
   String idempotencyKey,
   UriInfo uriInfo,
   SetupExecutable
preRpcExecutable,
   ProcessExecutable
rpcExecutable,
   PostProcessExecutable
postRpcExecutable)
   throws YourCustomException {
   try {
     IdempotencyRequest idempotencyRequest = createOrFindRequest(idempotencyKey, apiUri);
     Optional
responseOptional = findIdempotencyResponse(idempotencyRequest);
     if (responseOptional.isPresent()) {
       return responseOptional.get();
     }
     boolean isRetry = idempotencyRequest.isRetry();
     A requestObject = null;
     if (!isRetry) {
       requestObject = dbTransactionManager.execute(tc -> {
         A preRpcResource = preRpcExecutable.execute();
         updateIdempotencyResource(idempotencyKey, preRpcResource);
         return preRpcResource;
       });
     } else {
       requestObject = findRequestObject(idempotencyRequest);
     }
     R rpcResponse = rpcExecutable.execute(isRetry, requestObject);
     S response = dbTransactionManager.execute(tc -> {
       S postRpcResponse = postRpcExecutable.execute(isRetry, rpcResponse);
       updateIdempotencyResource(idempotencyKey, postRpcResponse);
       return postRpcResponse;
     });
     return serializeResponse(response);
   } catch (Throwable exception) {
     // Classify and handle retryable vs non‑retryable errors
   }
}

Handling Exceptions – Retry or Not?

Orpheus classifies exceptions into retryable (e.g., transient 5xx network errors) and non‑retryable (e.g., validation 4xx errors). Misclassification can lead to double payments or endless failures.

Client Responsibilities

Generate a unique idempotency key for each request and reuse it on retries.

Persist the key before invoking the service.

Cancel the key after a successful response.

Never change the request payload on retries.

Design exponential back‑off or jitter for automatic retries.

Choosing an Idempotency Key

Keys can be request‑level (random UUID) or entity‑level (e.g., "payment‑1234‑refund"). Request‑level keys are simpler and sufficient for most cases.

Lease (Expiration) for Each API Request

Each request acquires a row‑level lock on the idempotency key, granting a lease that expires after a configurable timeout, preventing duplicate processing during network failures.

Recording Responses

Responses are persisted to allow fast replay on retries, but the response table can grow large; periodic cleanup is required.

Using the Primary Database

All idempotency reads/writes are performed on the primary DB to avoid stale reads from replicas, which could cause duplicate payments due to replication lag.

Final Thoughts

Achieving eventual consistency in distributed payment systems requires added complexity. Orpheus provides a lightweight, generic solution that isolates idempotency logic, leverages Java lambdas for atomic transactions, and enforces strict error classification, enabling developers to focus on product features while maintaining data integrity.