Transactional Outbox in Go: Both Halves of Effectively-Once

Producer-side outbox plus consumer-side inbox dedup in Go and Postgres: ordering, cleanup, and running payment events at 1M+/day with zero double-processing.

Every article about the transactional outbox ends at the same place: the relay reads the table, publishes to Kafka, marks the row as sent, done. That’s half the pattern. On the payments/billing platform I work on, the half that actually prevented double-charging customers lives on the other side of the broker — in the consumer. This is a write-up of both halves: the outbox table everyone knows, and the inbox dedup, ordering decisions, cleanup, and operational grind that almost nobody writes about. We run this at 1M+ payment events a day with zero double-processing under retries and duplicate delivery. Here’s what it took.

TL;DR

A DB write and a Kafka publish are two independent writes. One of them will eventually fail alone, and both failure orderings cost you money.
Producer half: write the event into an outbox table in the same transaction as the state change. A polling publisher with FOR UPDATE SKIP LOCKED relays it to Kafka.
SKIP LOCKED with multiple publisher replicas silently destroys per-aggregate ordering. Decide whether you care before you scale to two pods.
Kafka’s “exactly-once” ends at the broker. Your consumer’s Postgres write or payment-provider call reruns on redelivery. So: consumer half — an inbox table with INSERT ... ON CONFLICT DO NOTHING inside the same transaction as the side effect.
Both tables grow forever. Cleanup is a day-one design decision, not a month-six firefight.
Outbox is choreography. When a flow needs multi-step rollback semantics, choreography runs out — that’s the Temporal article.

The dual-write bug loses money in both directions

The naive implementation of “save the payment and tell everyone about it” is two writes:

if err := repo.SavePayment(ctx, p); err != nil { // write 1: Postgres
    return err
}
if err := producer.Publish(ctx, evt); err != nil { // write 2: Kafka
    return err // ...and now what?
}

There is no transaction spanning Postgres and Kafka. Whichever order you pick, the gap between the two writes is a failure mode, and I’ve seen both variants in real code, not in a textbook:

Ordering A: commit DB first, publish second
┌─────────┐  1. COMMIT ok   ┌──────────┐
│ service │ ──────────────► │ Postgres │
└────┬────┘                 └──────────┘
     │ 2. publish
     ✖  crash / broker unavailable / pod OOM-killed
     ▼
  [ Kafka ]     payment saved, event LOST
                downstream never hears about the money

Ordering B: publish first, commit DB second
┌─────────┐  1. publish ok  ┌──────────┐
│ service │ ──────────────► │  Kafka   │
└────┬────┘                 └──────────┘
     │ 2. COMMIT
     ✖  serialization failure / deadlock / crash
     ▼
 [ Postgres ]   event announced, payment NEVER SAVED
                downstream acts on money that doesn't exist

Ordering A means a notification service never sends the receipt and the reporting pipeline undercounts revenue. Ordering B is worse: a consumer credits a balance for a deposit your own database rolled back. On a platform with a dozen+ payment-provider integrations, provider callbacks retry aggressively, so the window isn’t theoretical — it gets hit.

Wrapping write 2 in retries doesn’t fix this. Retries narrow the window; they can’t close it, because the process can die between the two writes. The only real fix is to make it one write.

The producer half: one transaction, one table

The outbox pattern turns two writes into one: the event goes into a table in the same database, in the same transaction as the state change. Postgres’s ACID guarantees do the coordination that no amount of retry logic can.

CREATE TABLE outbox (
    id            bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    aggregate_id  text        NOT NULL,  -- becomes the Kafka partition key
    topic         text        NOT NULL,
    event_type    text        NOT NULL,
    payload       jsonb       NOT NULL,  -- envelope includes a producer-side event UUID
    created_at    timestamptz NOT NULL DEFAULT now(),
    published_at  timestamptz,
    retry_count   int         NOT NULL DEFAULT 0
);

-- partial index: the publisher only ever scans unpublished rows
CREATE INDEX outbox_unpublished_idx ON outbox (id)
    WHERE published_at IS NULL;

Two details in that DDL are load-bearing. bigint, not serial: at 1M+ events a day an int4 sequence overflows in roughly five years, which is exactly long enough for everyone who knew about it to have left. And retry_count: a poison row — payload that fails serialization, a message over the broker’s size limit — will otherwise block its batch forever. You need an escape hatch to a dead-letter path.

The domain write becomes:

err := pgx.BeginFunc(ctx, pool, func(tx pgx.Tx) error {
    if err := savePayment(ctx, tx, p); err != nil {
        return err
    }
    _, err := tx.Exec(ctx, `
        INSERT INTO outbox (aggregate_id, topic, event_type, payload)
        VALUES ($1, $2, $3, $4)`,
        p.WalletID, "payments.events", "payment.captured", evt)
    return err
})

Either both rows exist or neither does. The dual-write bug is structurally impossible from this point on.

The publisher is a loop. Ours polls with FOR UPDATE SKIP LOCKED so multiple replicas can run without stepping on each other:

func (p *Publisher) pollOnce(ctx context.Context) (int, error) {
    tx, err := p.pool.Begin(ctx)
    if err != nil {
        return 0, err
    }
    defer tx.Rollback(ctx)

    rows, err := tx.Query(ctx, `
        SELECT id, topic, aggregate_id, payload
        FROM outbox
        WHERE published_at IS NULL
        ORDER BY id
        LIMIT 100
        FOR UPDATE SKIP LOCKED`)
    if err != nil {
        return 0, err
    }
    batch, ids, err := collectRecords(rows) // []*kgo.Record keyed by aggregate_id
    if err != nil {
        return 0, err
    }
    if len(ids) == 0 {
        return 0, tx.Commit(ctx)
    }

    // franz-go; idempotent producer and acks=all are the defaults
    if err := p.client.ProduceSync(ctx, batch...).FirstErr(); err != nil {
        return 0, fmt.Errorf("produce batch: %w", err) // rollback: rows stay unpublished
    }

    if _, err := tx.Exec(ctx,
        `UPDATE outbox SET published_at = now() WHERE id = ANY($1)`, ids); err != nil {
        return 0, err
    }
    return len(ids), tx.Commit(ctx)
}

The ordering of operations inside that function is the whole point. Mark rows published after the broker acks. If the publisher crashes mid-batch, the transaction rolls back, the locks release, and another replica picks the rows up again. That means the same event can be produced twice — and that’s fine, because we’re not pretending this is exactly-once delivery. It’s at-least-once, on purpose. The dedup lives downstream.

Batch size matters more than it looks. 100 rows per transaction is a sane starting point; the workable range is roughly 50–200. Go much larger and you hold row locks in a long transaction that also pins the xmin horizon, which blocks vacuum on a table that desperately needs vacuuming (more on that below).

If you’d rather not hand-roll this, Watermill’s Forwarder and libraries like oagudo/outbox exist and are fine. We hand-rolled anyway: on a payments platform you end up wanting control over batching, per-shard ordering, and metrics granularity that generic libraries don’t expose.

SKIP LOCKED gives you horizontal scale and quietly takes your ordering

Here’s the trap the tutorials skip. ORDER BY id inside one batch does not give you global ordering once you run two publisher replicas. Replica A grabs rows 1–100, replica B grabs 101–200 (SKIP LOCKED means B skips A’s locked rows), and B can reach Kafka first. Events 101–200 land before 1–100. Even with a single replica, sequence values commit out of order under concurrent writers — id order is not commit order. And if you were thinking of ordering by created_at: clock skew across pods makes that worse, not better.

Whether this matters depends on what ordering you actually need. Cross-aggregate ordering almost never matters. Per-aggregate ordering — all events for one wallet, one payment, in order — usually does. Your options, in ascending honesty:

Run a single publisher replica. Simple, ordered-enough in practice, and a single well-tuned poller sustains 1M+ events/day without breathing hard. Scale is rarely the real reason people run multiple relays; fear of a SPOF is, and Kubernetes restarting your one poller in seconds is usually an acceptable answer.
Shard the poll by aggregate: each replica polls WHERE (hashtext(aggregate_id) & 2147483647) % $n = $i. The bitmask matters: hashtext() returns a signed int4 that is negative for about half of all inputs, and in Postgres -5 % 3 = -2 — an unmasked % $n = $i predicate would leave every negative-hash aggregate unpublished forever. Per-aggregate order preserved, horizontal scale kept, complexity paid.
Use CDC, which preserves commit order for free.

Then the second half of ordering: the Kafka partition key. We key every record by aggregate_id. Same wallet → same partition → consumers see that wallet’s events in order. Key by event ID or leave the key nil and round-robin will interleave a single payment’s authorized/captured/failed events across partitions — and no consumer-side cleverness recovers that cheaply.

Polling vs Debezium: I run both, and neither wins outright

Our stack also runs Debezium CDC — PostgreSQL WAL into Kafka — so this comparison comes from operating both, not from reading both READMEs.

CDC’s pitch is real: no polling queries hammering the table, commit-order delivery for free, near-zero relay latency, and with the Outbox Event Router SMT you can even skip marking rows published. When it’s healthy, it’s the more elegant machine.

The ops bill is where it bites. A Debezium connector holds a logical replication slot, and a replication slot is a promise that Postgres will retain WAL until the connector reads it. Connector down over a weekend means WAL piles up until the disk fills, and a Postgres that runs out of disk is a much worse incident than a stale outbox. You mitigate with max_slot_wal_keep_size, heartbeats for quiet tables, and alerting on pg_replication_slots lag — but that’s a new class of failure your team now has to be fluent in, plus Kafka Connect itself as an extra distributed system to run and upgrade.

The polling publisher is a boring Go loop. Its failure modes are Postgres failure modes my team already knows: bloat, lock contention, vacuum. It adds a steady query load and a poll-interval’s worth of latency (typically low hundreds of milliseconds), and it costs you commit-order unless you shard.

My rule of thumb from running both: if your org already operates Kafka Connect competently and you need many tables streamed, Debezium earns its keep. If the outbox is your only CDC use case, the polling publisher’s boringness is a feature. For money-moving events, boring won.

Kafka’s exactly-once stops at the broker’s edge

Every few months someone reads about Kafka EOS and asks why we bother with dedup. Because the guarantee boundary is narrower than the marketing:

The idempotent producer (default since Kafka 3.0) dedups retries within a single producer session. Restart the publisher — which is exactly what happens in the crash-mid-batch case above — and the new session happily re-produces the same outbox row.
Kafka transactions make consume-transform-produce atomic within Kafka. The moment your consumer touches anything external — a Postgres UPDATE, a call to a payment provider — you’re outside the transaction. Consumer processes the message, writes to its DB, crashes before committing the offset: on rebalance, the message is redelivered and the side effect runs twice.

End-to-end, the honest contract is: at-least-once delivery, plus consumer-side dedup, equals effectively-once processing. The second half of that sentence is yours to build. I’ve watched consumers double-process on redelivery in exactly this pattern; it isn’t rare, it’s the default behavior of a correct at-least-once system.

The consumer half: the inbox table nobody writes about

The consumer-side fix is symmetric to the producer side and about as much code:

CREATE TABLE inbox (
    consumer     text        NOT NULL,   -- consumer group / logical consumer name
    message_id   uuid        NOT NULL,   -- producer-generated event ID from the envelope
    processed_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer, message_id)
);

The key choice is deliberate: a UUID minted by the producer and carried in the event envelope. Not the Kafka offset — offsets change if a topic is recreated or events are replayed from a rebuilt topic, which is precisely when you need dedup the most.

The consumer does the dedup check and the side effect in one database transaction:

func (c *Consumer) handle(ctx context.Context, rec *kgo.Record) error {
    var evt PaymentEvent
    if err := json.Unmarshal(rec.Value, &evt); err != nil {
        return c.toDLQ(ctx, rec, err) // poison message, don't block the partition
    }

    return pgx.BeginFunc(ctx, c.pool, func(tx pgx.Tx) error {
        tag, err := tx.Exec(ctx, `
            INSERT INTO inbox (consumer, message_id)
            VALUES ($1, $2)
            ON CONFLICT DO NOTHING`,
            c.name, evt.ID)
        if err != nil {
            return err
        }
        if tag.RowsAffected() == 0 {
            return nil // duplicate delivery: someone already processed it, commit no-op
        }
        return c.applyEffect(ctx, tx, evt) // credit balance, mark invoice, etc.
    })
}

Three things people get wrong here. First, check-then-insert (SELECT then INSERT) races under concurrent redelivery; ON CONFLICT DO NOTHING plus rows-affected is the atomic version. Second, the inbox insert and the side effect must share a transaction — dedup in Redis or a separate transaction reopens the exact gap you closed on the producer side. Third, this only covers side effects that live in the consumer’s own Postgres. If the side effect is an external call — say, a payout request to a provider — the inbox tells you whether you started, not whether you finished, and you need an idempotency key on the provider call itself. We put idempotency keys on every money-critical operation for exactly this reason; the inbox and the idempotency key cover different failure windows and you want both.

Put together, the full path of one payment event:

┌────────────┐ BEGIN                             ┌──────────┐
│ billing    │  INSERT payment                   │ Postgres │
│ service    │  INSERT outbox ────────────────►  │ (svc A)  │
└────────────┘ COMMIT  (atomic)                  └────┬─────┘
                                                      │ poll (SKIP LOCKED)
                                                ┌─────▼─────┐
                                                │ publisher │  at-least-once
                                                └─────┬─────┘
                                                      │ produce, key=aggregate_id
                                                ┌─────▼─────┐
                                                │   Kafka   │  (dupes possible)
                                                └─────┬─────┘
                                                      │ consume (redelivery possible)
┌─────────────────────────────────────┐              │
│ consumer          BEGIN             │◄─────────────┘
│  INSERT inbox ON CONFLICT DO NOTHING│   0 rows → skip, commit
│  apply side effect                  │   1 row  → process
│                   COMMIT  (atomic)  │
└─────────────────────────────────────┘
        = effectively-once processing

Duplicates are allowed everywhere in the middle. Correctness is enforced at both edges, atomically. That’s the design in one sentence, and it’s why we can say “zero double-processing” with a straight face: not because duplicates don’t happen, but because they’re absorbed.

Both tables grow forever unless you decide otherwise on day one

At 1M+ events/day, the outbox gains a million rows a day and the inbox gains a million per consumer. I’ve seen what an unbounded outbox turns into, and the failure is sneakier than “disk fills up.” The publisher’s hot query — the partial index scan over unpublished rows — degrades as dead tuples accumulate, because index-only scans fall back to heap visibility checks against millions of dead rows. Sadeq Dousti’s write-up documents the endgame: the same query going from 0.13ms to 18.5 seconds. And default autovacuum (autovacuum_vacuum_scale_factor = 0.2) won’t even fire until 20% of a huge table is dead.

The obvious fix — a nightly DELETE FROM outbox WHERE published_at < now() - interval '7 days' — makes it worse. Mass DELETE is what creates the dead tuples poisoning the hot index. Your realistic options:

DELETE + aggressive per-table autovacuum settings. Works at moderate volume if you tune autovacuum_vacuum_scale_factor way down for this table and delete in small batches. Simplest; where I’d start.
Time-based partitioning (pg_partman + pg_cron), retention by DROP PARTITION. Dropping a partition is a metadata operation — instant, no dead tuples, space reclaimed immediately.
Dousti’s trick: LIST-partition the outbox by published-status so unpublished rows live in a small hot partition and published rows land in a partition you periodically TRUNCATE.

The inbox needs the same treatment, with one constraint people miss: inbox retention must exceed your maximum replay window. If the topic retains events for 14 days and you keep inbox rows for 7, a replay on day 10 sails past your dedup and double-credits whatever it touches. Tie the two retention numbers together in one place, with a comment explaining why.

And one operational scar that applies to both tables: any index change on a hot outbox goes through CREATE INDEX CONCURRENTLY. A plain CREATE INDEX takes a lock that stalls every write — which on this table means stalling every domain transaction in the service, because the outbox insert is inside them. That’s the migration mistake that turns “add an index” into a partial outage. CONCURRENTLY can fail and leave an INVALID index behind; drop it and retry, don’t ignore it.

The metric that matters is insert-to-publish lag

Kafka consumer lag is the metric everyone already has. It’s the second most useful one here. The dashboard that actually catches outbox incidents early:

Outbox depth: count(*) WHERE published_at IS NULL. Should hover near zero.
Oldest unpublished age: now() - min(created_at) WHERE published_at IS NULL. This is your real end-to-end freshness bound. Alert on it; a wedged publisher shows up here minutes before any user notices.
Publisher batch metrics: rows per poll, produce latency, error rate, and retry_count > 0 row counts (poison candidates).
Postgres health for both tables: dead tuples from pg_stat_user_tables, last autovacuum time.
If you run Debezium: replication slot lag in bytes, alerted well below max_slot_wal_keep_size.

Every outbox problem I’ve dealt with announced itself in these numbers before it became visible downstream. The ClickHouse reporting service that consumes the same payment events off Kafka is also a decent canary — analytics noticing a gap is embarrassing but free monitoring, and keeping analytics on the Kafka side of the pipeline means its load never touches the transactional path.

Where the pattern runs out

Honest limits, because the outbox is not a universal answer.

Don’t build this for a modular monolith with one database (a transaction already covers you), for genuinely fire-and-forget events where a lost message costs nothing, or before you’ve accepted the tax: every event now costs an extra insert, a relay hop of latency, and two tables of care and feeding. Latency-sensitive request/response stays on gRPC in our stack; the outbox is for facts that must not be lost, not for questions that need answers now.

The deeper limit is architectural. The outbox is choreography: services emit facts and react to facts. That’s exactly right for fan-out — a payment happened, and notification, reporting, and reconciliation each do their thing independently. It stops being right when a flow needs multi-step writes across services with rollback semantics. We hit this with geo-distributed deposit flows: multi-step, cross-service, with timeouts and compensation when a step fails. Choreographed over topics, that becomes an implicit state machine smeared across services — no single place answers “where is this deposit stuck, and what unwinds if step 4 fails?” Retrofitting compensation onto event choreography means building a fragile orchestrator by accident, one dedup table and timeout topic at a time.

We moved those flows to a Temporal SAGA and kept the outbox for everything else. The rule I’d defend: outbox for facts, orchestration for processes. How that migration went — compensations, timers, versioning workflows in production — is its own article, linked below.

Before you ship this

How this system earned its numbers: payments microservices proof — the platform context behind the zero-double-processing claim.
When choreography ran out: a Temporal SAGA for geo-distributed deposits — the sibling article, picking up exactly where this one stops.