Mastering Distributed Transactions: From Blocking Retries to TCC and MQ

Introduction

In modern distributed systems and micro‑service architectures, inter‑service call failures are common. Ensuring data consistency and handling exceptions are essential challenges, with solutions such as blocking retries, 2PC/3PC, asynchronous queues, TCC compensation transactions, local message tables, and MQ transactions.

Blocking Retry

Blocking retry is a typical approach in micro‑services. Example pseudocode shows up to three retries of a request to B‑service; if all fail, an error is logged or propagated.

m := db.Insert(sql)

err := request(B-Service,m)

func request(url string,body interface{}){
  for i:=0; i<3; i++ {
    result, err = request.POST(url,body)
    if err == nil {
      break
    } else {
      log.Print()
    }
  }
}

The method can cause duplicate data, dirty data, and increased latency. It is suitable only when consistency requirements are low; otherwise additional mechanisms are needed.

Asynchronous Queue

Introducing a message queue decouples the service from the downstream call. After writing to the DB, a message is published to MQ for another service to consume.

m := db.Insert(sql)

err := mq.Publish("B-Service-topic",m)

Queue failures still leave a gap between DB write and MQ publish, so the same reliability issues as blocking retry remain.

TCC Compensation Transaction

TCC splits each service call into Try, Confirm, and Cancel phases, allowing compensation when a later step fails.

Try: check and reserve resources (e.g., inventory).

Confirm: commit the reservation.

Cancel: release the reservation if Try fails.

Example pseudocode for a shopping scenario:

m := db.Insert(sql)

aResult, aErr := A.Try(m)
 bResult, bErr := B.Try(m)
 cResult, cErr := C.Try(m)
 if cErr != nil {
   A.Cancel()
   B.Cancel()
   C.Cancel()
 } else {
   A.Confirm()
   B.Confirm()
   C.Confirm()
 }

Issues such as empty release, ordering, and network failures must be handled, often by idempotent retries or logging.

Local Message Table

The local message table, proposed by eBay, stores a message in the same transaction as business data. If subsequent steps succeed, the message is deleted; otherwise it remains for asynchronous retry.

Typical statuses are try and confirm. A background listener processes pending messages, ensuring eventual delivery.

Sample code shows inserting a message, publishing to MQ, and handling confirm/cancel logic.

MQ Transaction

Some MQ implementations (e.g., RocketMQ) support transactional messages. A message is first sent in a “prepare” state; after the business operation succeeds, a Confirm commits the message, otherwise a Cancel removes it.

Conclusion

For scenarios requiring strong consistency, additional mechanisms such as TCC, local message tables, or MQ transactions are necessary. TCC offers flexibility without coupling to a specific database but demands three APIs per service. Local message tables are simple and integrate well with existing services, while MQ transactions provide a centralized solution at the cost of limited MQ support and added latency.