How a Transactional Outbox Fixed Ghost Bugs and Duplicate Charges in Go Microservices

Story Begins: Nighttime Alert

A routine Friday night shift was interrupted by an urgent alarm and a flood of customer messages reporting duplicate order charges, a clear P0 production incident.

"Why was the order charged twice?" "I see duplicate orders." "Is the system broken?"

Root Cause Investigation

Initial checks confirmed that the payment gateway and external dependencies were operating normally. The issue was traced to the order‑svc (order service).

Payment gateway did not initiate duplicate payments.

External services were healthy.

The original order‑creation flow was:

Begin a database transaction.

Write order data.

Commit the transaction.

Publish an OrderCreated event to RabbitMQ for downstream services.

Problematic implementation:

// A risky implementation
func (s *orderService) CreateOrder(ctx context.Context, order *models.Order) error {
    // 1. Start transaction
    tx, err := s.db.Begin()
    if err != nil { return err }

    // 2. Insert order
    if err := tx.Insert(order); err != nil { return err }

    // 3. Commit transaction
    if err := tx.Commit(); err != nil { return err }

    // 4. Publish message
    event := mq.Event{ /* ... */ }
    if err := s.publisher.Publish(ctx, event); err != nil {
        // Data already committed, but message failed
        return err
    }
    return nil
}

The bug resides in step 4: tx.Commit() and publisher.Publish() are two independent distributed operations with no atomic guarantee, allowing a scenario where the transaction succeeds but the message appears to fail.

How the Ghost Bug Happens

The sequence that leads to duplicate charges:

User places an order.

Database transaction commits, creating Order A.

Service attempts to send OrderCreated to RabbitMQ.

The message reaches the broker, but the network acknowledgement times out.

Service interprets the timeout as a publish failure and returns an error.

Retry logic triggers a re‑execution of CreateOrder.

A second order, Order B, is inserted.

Downstream services receive two nearly identical OrderCreated events.

Coupons are redeemed twice, manifesting as duplicate charges.

One user action was executed twice.

Turning Point: Transactional Outbox

Because the database write and message publish cannot be made atomic, the design is changed to move the “send message” step inside the transaction as a local operation.

Store the intent to send a message in the database, then let a separate process reliably deliver it.

Transactional Outbox Workflow

Start a local transaction.

Create order data.

Do not send the MQ message directly.

Within the same transaction, insert an OutboxEvent record containing the event payload.

Commit the transaction.

An independent background service scans the outbox table.

It publishes the event to RabbitMQ.

After a successful publish, the outbox record is deleted.

Order data is guaranteed to exist.

The intent to send an event is guaranteed to be recorded.

Code Refactor

1. Define Outbox Model

// internal/shared/models/outbox.go
type OutboxEvent struct {
    ID         uuid.UUID `gorm:"primaryKey;type:uuid"`
    Exchange   string    `gorm:"not null"`
    RoutingKey string    `gorm:"not null"`
    Payload    []byte    `gorm:"type:bytea;not null"`
    CreatedAt  time.Time `gorm:"index"`
}

2. Rewrite Order Service (Write‑Only)

func (s *orderService) CreateOrder(ctx context.Context, order *models.Order) error {
    return s.db.RunInTransaction(func(tx db.TxTransaction) error {
        // 1. Insert order
        if err := tx.Insert(order); err != nil { return err }
        // 2. Marshal payload
        payload, _ := json.Marshal(order)
        // 3. Insert outbox record
        event := &models.OutboxEvent{ID: uuid.New(), Exchange: "orders.topic", RoutingKey: "order.created", Payload: payload}
        if err := tx.Insert(event); err != nil { return err }
        return nil
    })
}

3. Outbox Relay Service

type RelayService struct {
    db        db.Database
    publisher mq.Publisher
}

func (s *RelayService) Start() {
    go func() {
        for range time.NewTicker(10 * time.Second).C {
            s.processOutbox()
        }
    }()
}

The relay continuously scans the outbox table, publishes pending events to RabbitMQ, and removes successfully delivered records.

Results and Benefits

Data consistency : Order data and the intent to emit an event are stored atomically.

Reliable delivery (at‑least‑once) : Even if the message broker is temporarily unavailable, the event is persisted and retried.

Separation of concerns : Business logic no longer handles message‑sending retries or error handling.

The complete implementation can be examined in the open‑source repositories:

GitHub: https://github.com/louis-xie-programmer/easyms.golang

Gitee: https://gitee.com/louis_xie/easyms.golang