A payment provider sends a success callback. Your service credits the wallet, then crashes before recording that it did. The provider, seeing no ACK, resends the callback. Now the customer has been credited twice, the ledger disagrees with the wallet, and someone gets to spend an afternoon writing a reconciliation script. On the payments/billing platform I work on, that failure mode got materially worse the day we moved wallet services out into geographic regions — because “credit the wallet” stopped being a local call and became a gRPC hop across a WAN. This is how we made cross-region deposits effectively exactly-once with a 6-step SAGA on Temporal, in Go, with real timeout and retry numbers rather than tutorial defaults.
TL;DR
- Wallet/bonus services live in regions (eu/asia/default); payment stays in core. A deposit is now a distributed transaction across a WAN, orchestrated by a Temporal SAGA.
- Six steps, LIFO compensations. Compensation stacks run under a disconnected context and retry without limit.
- Every side-effecting step carries a deterministic idempotency key derived from the saga:
operation_guid = saga_idfor the wallet credit,"revert-" + operation_guidfor the revert,saga_idas the Kafka producer key. - The FX rate is fetched once and replayed from Temporal event history on every retry. A retry can never change the amount. This is the single most important line in the design.
- Kafka sits in front as the durable delivery layer: the callback consumer retries until the SAGA is accepted, so the system survives Temporal downtime.
- Failure after the wallet credit is a forward retry, not a rollback. Rolling back money that already moved is how you create the incident you were trying to prevent.
Direct calls stopped being safe when wallets moved to regions
The original flow was simple: the payment service receives a provider callback, calls the wallet service over gRPC in the same cluster, updates the payment row, produces a Kafka event for the bonus consumer. Three side effects, one process, low latency. When it broke mid-sequence, the blast radius was small and the retry logic was a couple of if statements.
Then wallets and bonuses moved into regions for data-locality, and the same sequence became this:
provider ──callback──> payment-svc [core]
│
│ gRPC over WAN
v
wallet-svc [eu] balance: +100 OK
│
X payment-svc crashes here:
payment row still "pending",
no Kafka event produced
provider resends callback (no ACK seen)
│
v
wallet-svc [eu] balance: +200 <- double creditThree properties of the new world made the naive approach untenable:
- The WAN is in the transaction. A cross-region gRPC call fails for reasons a local call never did: transient routing, regional degradation, an entire region being down. Each failure leaves you mid-sequence.
- Partial completion is now the common case, not the edge case. With three side effects across two failure domains, “step 2 done, step 3 not” is a state you plan for from the start.
- The callback consumer must not block. Provider callbacks arrive via Kafka. If the consumer blocks on a cross-region call, one slow region backs up the callback lag for everyone.
We already ran a transactional outbox for reliable event publication (see the sibling article), and it stayed. But an outbox gives you at-least-once delivery of one event. It does not give you a multi-step transaction with per-step rollback across regions. Choreography — each service reacting to events — would have scattered the deposit’s state machine across three codebases with no single place to answer “where is this deposit and why is it stuck?”. For a money flow with six steps and compensation semantics, we wanted orchestration with durable state. That’s Temporal’s exact shape.
The shape of the system
Two components, one workflow:
- billing-saga-core (core): Temporal worker plus a gRPC server. Owns the deposit workflow and all core-side activities.
- billing-saga-region (one per region): executes wallet operations locally, against the region’s wallet service.
CORE REGION (eu / asia)
┌────────────────────────────────────────┐ ┌─────────────────────────────┐
│ Kafka (provider callbacks) │ │ │
│ │ │ │ billing-saga-region │
│ v │ │ worker polls task queue │
│ payment-svc ──gRPC (fire-and-forget)──│ │ billing-saga-region-eu │
│ returns saga_id immediately │ │ │ │
│ │ │ │ v │
│ v │ │ wallet-svc (local gRPC) │
│ billing-saga-core ◄──► Temporal ──────┼WAN─┼──────────┘ │
│ (workflow + core server │ │ │
│ activities) PG persistence │ └─────────────────────────────┘
│ ES visibility │
│ │ │
│ v │
│ Kafka (deposit-events → bonus) │
└────────────────────────────────────────┘Routing works in two layers. Requests with a JWT carry a region claim (default/eu/asia; a missing claim means default, for backward compatibility). Provider callbacks arrive via Kafka with no JWT, so a dedicated Region Service resolves GetCustomerRegion(customer_guid) -> region — Redis in front of Postgres, low-latency, because it sits on the hot path of every callback.
The payment service stays thin. Default region keeps the old direct wallet call — no reason to pay orchestration overhead for a local call. Any other region: a fire-and-forget gRPC call to saga-core, which returns a saga_id immediately and never blocks the callback consumer.
Geo-routing inside the workflow is just task queues. Core activities run on billing-saga-core; wallet activities are scheduled onto billing-saga-region-{name}, and only the worker deployed in that region polls that queue. The workflow code doesn’t know or care where the worker physically is — the queue name is the routing decision.
Six steps, and what each compensation means in money
The deposit SAGA has six steps. Compensations run LIFO, and that ordering matters: each step’s compensation assumes the steps after it have already been undone.
forward ─────────────────────────────────────────────────────>
1 FindOrCreatePayment [core] comp: mark cancelled
2 ParseCallbackStatus [core, local] read-only, no comp
3 ConvertCurrency [core] rate pinned in history, no comp
4 UpdateWalletBalance [region] key: saga_id
comp: RevertWalletBalance
key: "revert-"+saga_id
5 UpdatePaymentStatus [core] comp: mark failed
6 ProduceDepositEvent [core] Kafka, producer key: saga_id
<───────────────────────────────────────────── compensate (LIFO)
fail at 6: 5' mark failed → 4' revert wallet → 1' cancel
fail at 5: 4' revert wallet → 1' cancel
fail at 4: 1' cancel (money never moved)In money terms:
- FindOrCreatePayment — look up the payment by
external_id; create it if this is the webhook-only flow, where the callback is the first we hear of the deposit. Compensation: mark it cancelled. No money involved yet, but an orphaned “pending” row is a support ticket. - ParseCallbackStatus — parse the provider’s status. Read-only, runs as a local activity (it’s pure computation; a full activity round-trip through the server would be waste). No compensation because there is nothing to undo.
- ConvertCurrency — get the FX rate from the currency service and compute the credit amount. More on this below; it’s the centerpiece.
- UpdateWalletBalance — the gRPC deposit into the regional wallet. This is where money moves. Compensation is
RevertWalletBalance, a distinct operation with its own key — not a “delete”, because the wallet keeps both entries in its ledger. - UpdatePaymentStatus — set the payment successful, attach provider and conversion logs and the transaction number. Compensation: mark failed.
- ProduceDepositEvent — publish to the
deposit-eventstopic for the bonus consumer. No compensation of its own; if it exhausts retries, the LIFO stack unwinds everything above it.
Why LIFO and not “undo in any order”? Because step 5’s compensation (mark failed) is only truthful if step 6’s event never became visible to consumers, and step 4’s revert should happen while the payment row still reflects reality. Unwinding in reverse dependency order means every compensation runs against the state its forward step created.
The workflow code, with the real numbers
Trimmed but honest Go. The numbers are the production numbers, not placeholders.
func DepositWorkflow(ctx workflow.Context, in DepositInput) error {
sagaID := workflow.GetInfo(ctx).WorkflowExecution.ID
// Forward activities: fail fast, bounded retries.
ao := workflow.ActivityOptions{
ScheduleToCloseTimeout: 30 * time.Second,
StartToCloseTimeout: 10 * time.Second,
// Our activities call activity.RecordHeartbeat; drop this if yours don't,
// or a legal 6-10s call will die on heartbeat timeout instead.
HeartbeatTimeout: 5 * time.Second,
RetryPolicy: &temporal.RetryPolicy{
InitialInterval: time.Second,
BackoffCoefficient: 5.0, // waits 1s, then 5s (3 attempts total)
MaximumAttempts: 3,
},
}
coreCtx := workflow.WithActivityOptions(ctx, ao)
regionAO := ao
regionAO.TaskQueue = "billing-saga-region-" + in.Region
regionCtx := workflow.WithActivityOptions(ctx, regionAO)
// LIFO compensation stack. Runs on a disconnected context so it
// still executes after cancellation; compensations retry forever.
var comps []func(workflow.Context) error
unwind := func() {
dc, _ := workflow.NewDisconnectedContext(ctx)
dc = workflow.WithActivityOptions(dc, workflow.ActivityOptions{
StartToCloseTimeout: 10 * time.Second,
RetryPolicy: &temporal.RetryPolicy{MaximumAttempts: 0}, // unlimited
})
for i := len(comps) - 1; i >= 0; i-- {
_ = comps[i](dc)
}
}
// 1. Find or create the payment.
var pay Payment
if err := workflow.ExecuteActivity(coreCtx,
a.FindOrCreatePayment, in.ExternalID).Get(ctx, &pay); err != nil {
return err
}
comps = append(comps, func(c workflow.Context) error {
return workflow.ExecuteActivity(c,
a.MarkPaymentCancelled, pay.GUID).Get(c, nil)
})
// 2. Parse provider status (local activity: pure computation).
lctx := workflow.WithLocalActivityOptions(ctx, workflow.LocalActivityOptions{
StartToCloseTimeout: 10 * time.Second,
})
var st CallbackStatus
if err := workflow.ExecuteLocalActivity(lctx,
a.ParseCallbackStatus, in.RawCallback).Get(ctx, &st); err != nil {
unwind()
return err
}
// wait-confirm flow: park until the psp-callback signal arrives.
// WorkflowExecutionTimeout (1h) bounds the wait.
if st.WaitConfirm {
var cb ProviderCallback
ch := workflow.GetSignalChannel(ctx, "psp-callback")
ch.Receive(ctx, &cb)
st = cb.Status
}
// 3. FX conversion. Executes once; the result lives in event
// history and is replayed on every retry. Never re-fetched.
var amt ConvertedAmount
if err := workflow.ExecuteActivity(coreCtx,
a.ConvertCurrency, pay.Currency, in.WalletCurrency,
pay.Amount).Get(ctx, &amt); err != nil {
unwind()
return err
}
// 4. Credit the regional wallet. operation_guid = saga_id.
if err := workflow.ExecuteActivity(regionCtx,
a.UpdateWalletBalance, WalletOp{
OperationGUID: sagaID,
CustomerGUID: in.CustomerGUID,
Amount: amt.Value,
}).Get(ctx, nil); err != nil {
unwind()
return err
}
comps = append(comps, func(c workflow.Context) error {
// Same regional task queue, but MaximumAttempts: 0 — the revert
// must eventually land, so it inherits the unlimited-retry policy.
rc := workflow.WithActivityOptions(c, regionAOForComp(in.Region))
return workflow.ExecuteActivity(rc,
a.RevertWalletBalance, WalletOp{
OperationGUID: "revert-" + sagaID,
CustomerGUID: in.CustomerGUID,
Amount: amt.Value,
}).Get(rc, nil)
})
// 5. Mark the payment successful (idempotent UPDATE).
if err := workflow.ExecuteActivity(coreCtx,
a.UpdatePaymentStatus, pay.GUID, st, amt).Get(ctx, nil); err != nil {
unwind()
return err
}
comps = append(comps, func(c workflow.Context) error {
return workflow.ExecuteActivity(c,
a.MarkPaymentFailed, pay.GUID).Get(c, nil)
})
// 6. Publish for the bonus consumer (idempotent producer, key=saga_id).
if err := workflow.ExecuteActivity(coreCtx,
a.ProduceDepositEvent, sagaID, pay, amt).Get(ctx, nil); err != nil {
unwind()
return err
}
return nil
}Notes on the numbers, because defaults are where tutorials lie to you:
- StartToClose 10s, ScheduleToClose 30s. A wallet credit that takes longer than 10 seconds is not slow, it’s broken. ScheduleToClose caps the whole retry envelope so a dead region fails the step in half a minute instead of hanging.
- 3 attempts, backoff 1s then 5s. Enough to ride out a transient gRPC blip; not enough to mask a real outage. A payment saga that silently retries for an hour is worse than one that fails loudly inside its 30-second envelope.
- Compensations retry without limit (
MaximumAttempts: 0). Forward progress is optional; undoing a wallet credit is not. If the revert can’t land, we want the activity red on the dashboard until either the region recovers or a human intervenes — not a swallowed error and a wallet that’s out of balance. - WorkflowExecutionTimeout 1h bounds everything, including the wait-confirm signal wait. No saga is ever “stuck” for longer than an hour by construction.
Duplicate callbacks are handled before the workflow even starts, by Workflow ID:
run, err := c.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
ID: fmt.Sprintf("deposit-%s-%s", cb.ExternalID, cb.EventType),
TaskQueue: "billing-saga-core",
WorkflowExecutionTimeout: time.Hour,
WorkflowIDReusePolicy: enumspb.WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE,
TypedSearchAttributes: temporal.NewSearchAttributes(
saPaymentGUID.ValueSet(cb.PaymentGUID),
saCustomerGUID.ValueSet(cb.CustomerGUID),
saRegion.ValueSet(region),
),
}, DepositWorkflow, input)The provider resends the callback? Same external_id, same event_type, same Workflow ID — Temporal rejects the duplicate start. Deduplication costs zero application code. The search attributes set at start (payment_guid, customer_guid, region) — with amount and saga_status upserted as the workflow progresses — mean support can find any deposit in the Temporal UI without touching a database.
The FX rate is pinned in event history, and that is the point
A deposit arrives in EUR, the wallet is in another currency, and ConvertCurrency fetches the rate from a currency service. Rates move constantly. Now consider: the wallet credit at step 4 times out, retries, and succeeds 26 seconds later. If anything in that retry path re-fetched the rate, the amount credited could differ from the amount we’re about to log on the payment and publish to Kafka. The customer was credited X; the books say Y — and someone is back to writing that reconciliation script.
Temporal’s execution model closes this hole for free — if you respect it. An activity result, once recorded in event history, is replayed on every subsequent workflow replay, never re-executed. ConvertCurrency runs once, its result becomes a fact in history, and every retry of every later step — and every worker crash and replay of the entire workflow — sees the identical amount. The rate is fixed at first execution and never moves again.
The corollary is the classic determinism rule with money attached: never fetch a rate, read a clock, or generate an ID inline in workflow code. Do it in an activity, or derive it deterministically from workflow state. The moment someone “simplifies” the conversion into a plain function call inside the workflow, replays stop being deterministic and retries start changing amounts. We treat this as a review-blocking rule, and it’s the first thing I check in any workflow diff.
Idempotency keys are the whole trick
Temporal gives you effectively-once workflow semantics, but activities are at-least-once: a worker can execute the wallet credit, then die before reporting completion, and Temporal will schedule the activity again. Exactly-once effects therefore have to come from the target systems, and that means deterministic keys:
- UpdateWalletBalance:
operation_guid = saga_id. The saga ID is generated once, before any side effect, and is stable across every retry and replay. The wallet service dedupes on it, so at-least-once execution collapses into exactly-one credit. Deriving the key from anything generated during execution (a timestamp, a random UUID inside the activity) would silently reintroduce double credits. - RevertWalletBalance:
"revert-" + operation_guid. The revert is its own idempotent operation with its own key — retrying the compensation forever is only safe because replaying it is harmless. - UpdatePaymentStatus: an idempotent UPDATE keyed by
payment_guid+ status:
UPDATE payments
SET status = 'successful',
transaction_number = $2,
provider_log = $3,
conversion_log = $4,
updated_at = now()
WHERE payment_guid = $1
AND status <> 'successful';- ProduceDepositEvent: Kafka idempotent producer, keyed by
saga_id, so the bonus consumer can dedupe even if the produce activity retries after a successful-but-unacknowledged send.
One saga ID, threaded through every side effect. When something goes wrong, that single ID joins the Temporal history, the wallet ledger, the payment row, and the Kafka event.
Kafka in front means Temporal is allowed to go down
An orchestrator is a dependency, and dependencies fail. We deliberately did not put Temporal on the callback ingestion path. Provider callbacks land in Kafka; the consumer’s job is only to get the saga accepted — and it retries until it succeeds. If Temporal is down for ten minutes, callbacks accumulate in the topic with Kafka’s durability guarantees, and drain when it comes back. No lost deposits, no provider-side retry storms hitting a dead endpoint. Temporal orchestrates the money movement; Kafka guarantees the request to move money is delivered at all. The system tolerates orchestrator downtime by never making the orchestrator the front door.
The failure modes we designed for
No war story here — this system hasn’t had a public incident to narrate, and I’m not going to invent one. What I can share is the failure-mode matrix we designed against, which is where the actual lessons live:
- Region down. Wallet activities on
billing-saga-region-eutime out (ScheduleToClose 30s), the saga fails, compensations run, an alert fires. Bounded, loud, clean. The alternative — infinite retry against a dead region — hides an outage inside “pending” sagas. - Wallet gRPC down, region up. 3 attempts inside the 30-second ScheduleToClose envelope, then compensation. Transient blips get absorbed; real failures surface fast.
- Failure after the wallet credit. This is the case that separates designs. Money has moved. Steps 5 and 6 are idempotent bookkeeping. The correct move is a forward retry, not a rollback: retry marking the payment and publishing the event until they land. Reverting a credit the customer may already see — because a Kafka produce hiccuped — converts a retryable glitch into a customer-visible balance drop. Rollback is for “the deposit cannot complete”, not “step 6 is being slow”.
- Duplicate callbacks. Handled at the Workflow ID layer, before any business code runs.
- Stuck saga. Structurally impossible past one hour: WorkflowExecutionTimeout force-closes the run as timed out, and an alert fires on that terminal state. Note what does not happen: an execution timeout kills the workflow without running compensations — which is exactly why every step is idempotent and keyed by the saga ID, and why a timed-out saga lands in a human review queue instead of pretending it cleaned up after itself.
Transport across the WAN: Temporal long-poll vs NATS request/reply
Core-to-region transport got two designs on the whiteboard.
Option A — regional Temporal worker. The region runs a plain Temporal worker that long-polls the core Temporal server over gRPC across the WAN, on its billing-saga-region-{name} queue. Wins: one system, one execution model; retries, timeouts, and history come for free; connections are outbound from the region, which makes firewall and NAT traversal a non-issue; monitoring is the same Temporal metrics everywhere. Cost: worker-to-server traffic crosses the WAN, so link instability shows up as schedule-to-start latency you must watch.
Option B — NATS request/reply. Subjects billing.{region}.wallet.{update|revert|get}, a stateless subscriber in each region, the workflow’s regional activity becomes a core-side activity doing a NATS request. Wins: a lean transport tuned for exactly this hop; the regional footprint is a trivial subscriber with no Temporal dependency. Cost: a second messaging system to deploy, secure, and monitor in every region; retry and timeout semantics now live in two places; you’ve reinvented a slice of what the task queue already gave you.
The honest summary: option A keeps operational complexity flat and pushes risk into WAN link quality; option B buys transport independence at the price of running more infrastructure. If you already operate NATS everywhere, B is defensible. If Temporal is the only new system in the picture, adding a second one just to avoid a long-poll over the WAN is complexity without a payoff.
What we rejected, and when you should reject Temporal
- Hand-rolled Kafka saga. State machine tables, timeout sweepers, compensation dispatchers — you end up writing a worse Temporal with none of the replay guarantees and all of the maintenance.
- Outbox + choreography alone. Great for “this event must be published”; wrong shape for “these six steps must complete or be undone in order”. The deposit’s state ends up implicit, smeared across consumers.
- Two-phase commit. Locks held across a WAN, a coordinator as a single point of failure, and half our participants (Kafka, external-ish wallet APIs) don’t speak XA anyway.
- Lightweight Temporal-like libraries. Less to operate, but the persistence, visibility, and replay tooling is exactly the part you need when money is involved. That’s the part the lightweight options cut.
And the honest inverse — when Temporal is overkill: single-region flows where one database transaction covers all side effects; flows where the only requirement is reliable event publication (the outbox alone is simpler and cheaper); teams without the capacity to operate another stateful system, because a Temporal cluster with PostgreSQL persistence and Elasticsearch visibility is real infrastructure with real failure modes of its own. We kept the default region on the direct wallet call for exactly this reason: orchestration where the WAN demands it, boring local calls where it doesn’t.
Before you ship this
- Every side-effecting activity has a deterministic idempotency key derived from the saga ID, and the target system actually dedupes on it. Test the dedup, not just the happy path.
- Anything non-deterministic — FX rates, clocks, generated IDs — lives in activities, never inline in workflow code. Verify with replay tests against exported histories in CI.
- Forward retries are bounded (attempts + ScheduleToClose); compensation retries are unlimited and alerted on.
- Failure-after-money-moved is explicitly a forward retry, and your team can articulate why.
- WorkflowExecutionTimeout is set — and your team knows it terminates the run without running compensations, and who gets paged when it fires.
- Workflow IDs encode natural dedup keys (
external_id+ event type) with a reject-duplicate reuse policy. - The ingestion path survives orchestrator downtime — a durable queue in front, with a consumer that retries until the workflow is accepted.
- Search attributes cover the fields support will actually query: payment ID, customer ID, region, amount, saga status.
- You’ve written down what happens when a compensation can’t land after N hours, and a human is in that loop.
Related
- Sibling article: The transactional outbox pattern in the same platform — the reliable-publication layer this saga builds on.
- Proof page: Temporal SAGA for geo-distributed billing — architecture notes and outcomes for this system.