Architecture decision · fintech
Geo-distributed billing with a Temporal SAGA
Backend / Platform GoTemporalgRPCKafkaNATSPostgreSQL
callback ─► core · Temporal SAGA ──task queue──► region wallet find/create → parse → FX (replayed) → deposit → status → event any failure ⇒ LIFO compensations · Kafka keeps callbacks durable
Problem
A distributed payments/billing platform needed to process money-critical deposits across regions with exactly-once guarantees — surviving region outages, network partitions, and duplicate provider callbacks — without a fragile hand-rolled coordinator or two-phase commit.
Approach
Modelled the deposit as a Temporal SAGA — a six-step workflow with LIFO compensations. Wallet operations run in-region while orchestration stays in core; requests route by region via task queues and gRPC fire-and-forget, and the core↔region transport was designed and compared across a regional Temporal worker over WAN versus a NATS request/reply path. Every activity carries its own idempotency key, the FX rate is fixed on first execution and replayed from event history on retries, and provider callbacks arrive over Kafka so delivery survives a Temporal outage.
Result
Cross-region deposits that are effectively exactly-once — a failed step triggers compensations, a hung workflow is bounded by a timeout, duplicate callbacks dedupe on workflow ID, and a downed region degrades safely with alerts instead of losing or double-crediting money.
Evidence
Described in work history (Korvax, 2022–present). Design trade-offs — the core↔region transport comparison, compensation semantics, and the failure-mode matrix — are discussable on a call.
Available for: private walkthrough
The workflow is the coordinator: Temporal persists each step, so a crash resumes from the last completed activity instead of re-running side effects. Idempotency on every activity plus LIFO compensations are what make “credit this deposit exactly once” true across regions — even when callbacks are at-least-once and a whole region can vanish mid-flight.