--- title: "Multi-Region Active-Active for a Payments API" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/post/multi-region-active-active-payments --- ![Blog post image for Multi-Region Active-Active for a Payments API - How a money-movement API was taken active-active across two AWS regions with idempotency keys, conflict-free replication, and a tested RTO and RPO, so a full regional outage never double-charges a customer or loses a committed payment.](/_astro/hero.RPbRRCdE_22Muqg.webp) [Home](/)›[Case studies](/case-studies)›[All Categories](/case-studies/categories)›[Architecture](/case-studies/categories/architecture) Case studies [Prev in ArchitectureMigrating a Monolith to Kubernetes Without a Big-Bang Cutover](/case-studies/post/monolith-to-kubernetes-strangler-migration) [Architecture](/case-studies/categories/architecture)[Cloud Computing](/case-studies/categories/cloud-computing)[Reliability](/case-studies/categories/reliability) # Multi-Region Active-Active for a Payments API [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 16 Aug 2026Updated: 16 Aug 202606 Mins read09 Mins listen [Markdown for AI(opens in a new tab)](/post/multi-region-active-active-payments/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR How a money-movement API was taken active-active across two AWS regions with idempotency keys, conflict-free replication, and a tested RTO and RPO, so a full regional outage never double-charges a customer or loses a committed payment. Series [Cloud Platforms & Architecture](/series/cloud-platforms--architecture)1/1 All posts in this series (1) Case Studies1 1. [Multi-Region Active-Active for a Payments APIYou are here](/case-studies/post/multi-region-active-active-payments) ### Multi-Region Active-Active for a Payments API Contents [Impact](#impact)[The problem](#the-problem)[Constraints](#constraints)[Architecture](#architecture)[Implementation](#implementation)[Results](#results)[Lessons](#lessons)[Frequently Asked Questions](#frequently-asked-questions)[References](#references) A payments API that moves real money had been running comfortably in a single AWS region for years. It was reliable until the day it was not: a regional control-plane incident took the whole service offline for a few hours, and there was no second region to fail over to. For most products that is an outage. For a money-movement API it is stuck settlements, angry partners, and a compliance conversation. The mandate that came out of that incident was simple to say and hard to build: survive the loss of an entire region without losing a committed payment or charging anyone twice. This is the story of taking that API active-active across two regions. The interesting part is not the traffic routing, which is close to a solved problem. The interesting part is the money: making retries safe, keeping two live databases honest, and proving the failover actually works instead of trusting a diagram. ## [Impact](#impact) 0double charges in production 0AWS regions, both live < 0 minmeasured RTO 0%game-days passing Once the second region went live, a full regional failure stopped being an incident and became a drill. During the quarter after cutover the primary region had two brief degradations, and in both cases traffic shifted to the healthy region inside the target window with no customer-visible errors and, most importantly, no duplicate settlements. The number the finance and risk teams cared about was the double-charge count, and it stayed at zero. That is not because failures stopped happening. It is because every write path was made idempotent and every retry, whether from a client, a load balancer, or a queue redelivery, converges on the same result. ## [The problem](#the-problem) A single-region payments API has two failure modes that a diagram tends to hide. The first is total loss of the region, which is rare but catastrophic and completely outside your control. The second, and the one that actually bites during a failover, is the retry storm: when a region gets shaky, every client, proxy, and queue in the system starts retrying, and if those retries are not idempotent, you turn one payment into several. The business could tolerate a couple of minutes of elevated latency during a failover. It could not tolerate a lost payment that a customer had already seen succeed, and it absolutely could not tolerate charging a card twice. So the real problem was not “run in two regions.” It was “make every money-touching operation safe to repeat, then run in two regions.” ## [Constraints](#constraints) The design had to fit inside some hard limits. Payments are regulated, so data residency rules meant certain records could not leave their region of origin, which ruled out a naive single global write master. The team ran on Kubernetes (EKS) and Aurora PostgreSQL already, so the solution had to build on those rather than introduce an exotic new datastore. And the failover had to be measurable: leadership wanted a specific RTO and RPO written down and proven, not a hand-wave. There was also a people constraint. On-call engineers needed a failover they could trust at 3am without a runbook full of manual database promotion steps, because manual steps under pressure are how a recoverable incident becomes a data-loss incident. ## [Architecture](#architecture) Both regions run the full stack and take live traffic. Route 53 uses latency-based routing with health checks so users hit the closest healthy region, and it fails a region out automatically when its health check trips. Each region has its own ALB, API pods on EKS, an idempotency store, and a database. Route 53 latency routing sends traffic to the nearest healthy region. Each region runs the full stack (ALB, EKS, idempotency store, database). The idempotency store is a DynamoDB global table and the system of record is Aurora Global Database with sub-second replication. Two decisions carry the whole design. The idempotency store is a DynamoDB global table, replicated multi-active across both regions, so a key claimed in one region is visible in the other within about a second. The system of record is Aurora Global Database: a writer in the primary region with sub-second physical replication to the secondary, where a reader can be promoted to writer during a failover in roughly a minute. Committed transactions replicate fast enough that the recovery point stays effectively at zero for anything the customer already saw succeed. The failover path is deliberately boring. A health check trips, DNS shifts, Aurora promotes the secondary, and in-flight retries replay with their idempotency key. When Region A degrades, the Route 53 health check trips, Aurora promotes the Region B reader to writer in about a minute, and in-flight retries replay with their idempotency key. RTO stays under two minutes and RPO stays near zero for committed transactions. ## [Implementation](#implementation) The idempotency key does the real work here. Every payment request carries an `Idempotency-Key` header. Before doing any work, the API does a conditional write into the idempotency store to claim that key. If the key already exists, the stored result is returned as-is and no charge happens. If the claim succeeds, the API runs the charge inside a database transaction, records the result under the key with a TTL, and returns it. Any retry, from any region, with the same key gets the same answer. The API claims the idempotency key before charging and stores the result against it. A retry, in the same region or a different one after failover, finds the key already present and returns the original result instead of charging the card twice. The subtle bug to avoid is claiming the key and then crashing before the result is stored, which would leave a claimed-but-unfinished key that blocks the retry forever. The fix is to store an in-progress marker at claim time and let the retry either return the finished result or safely resume, with the transaction as the source of truth for whether the money actually moved. Events flowing out to downstream systems (ledgers, notifications) use a transactional outbox, written in the same transaction as the payment, so an event is emitted exactly once per committed payment and consumers dedupe on the same key. That keeps the two regions from emitting conflicting events for the same operation. ## [Results](#results) Across the first quarter live, the primary region degraded twice. Both times Route 53 shifted traffic and Aurora promoted the secondary well inside the two-minute RTO target, and customers saw a short latency bump rather than errors. No payment was lost and nothing was charged twice, which was the entire point. The less glamorous result was operational confidence. Because the failover is automatic and every write is idempotent, on-call stopped treating a regional wobble as an emergency. The monthly game-day, where a region is deliberately failed out in production-like conditions, went from a nerve-wracking event to a routine check with a green result. ## [Lessons](#lessons) The biggest lesson is that active-active is a data problem wearing a networking costume. Getting traffic to two regions is easy; keeping two live copies of money honest is the hard part, and idempotency is what makes it tractable. If you cannot safely repeat every write, no amount of clever routing will save you during a failover. The second lesson is that an RTO and RPO you have not tested are just wishes. The game-days repeatedly surfaced small issues (a too-aggressive health-check threshold, a client that did not send idempotency keys on one endpoint) that no diagram would have caught. Failover is a feature, and like any feature it has bugs until you exercise it. ## [Frequently Asked Questions](#frequently-asked-questions) Active-passive keeps a warm standby that only takes traffic during a failover, which means the standby path is rarely exercised and tends to rot. Active-active runs real traffic through both regions all the time, so the failover path is the same path you use every day. It costs more, but for a money-movement API the confidence that the second region actually works is worth it. Every payment request carries a client-generated key. The API claims that key in a globally replicated store before charging, and stores the result against it afterward. If a retry arrives, in the same region or a different one after failover, the key is already present and the original result is returned without charging again. The key, not the region, is what guarantees exactly-once. RTO (recovery time objective) is how long the service can be unavailable before it is back, which here is the couple of minutes it takes DNS to shift and Aurora to promote a writer. RPO (recovery point objective) is how much committed data you can lose, which here is effectively zero because idempotency writes are synchronous and database replication lag stays under a second for committed transactions. It constrains it. Records that legally must stay in their region of origin are not globally writable, so the design keeps the system of record regional (a promotable writer per region) rather than a single global write master. The globally replicated piece is the idempotency store, which holds keys and results, not the regulated ledger data. ## [References](#references) - [Amazon Aurora Global Database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html) - [Amazon DynamoDB global tables](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GlobalTables.html) - [Making retries safe with idempotent APIs (AWS Builders’ Library)](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) - [Amazon Route 53 health checks and DNS failover](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html) - [Transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html) Was this useful? ## Tags [#Multi Region](/case-studies/tags/multi-region)[#Active Active](/case-studies/tags/active-active)[#Payments](/case-studies/tags/payments)[#Idempotency](/case-studies/tags/idempotency)[#Failover](/case-studies/tags/failover)[#High Availability](/case-studies/tags/high-availability)[#RTO](/case-studies/tags/rto)[#RPO](/case-studies/tags/rpo) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fmulti-region-active-active-payments "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Multi-Region%20Active-Active%20for%20a%20Payments%20API&url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fmulti-region-active-active-payments "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fmulti-region-active-active-payments&title=Multi-Region%20Active-Active%20for%20a%20Payments%20API&summary=How%20a%20money-movement%20API%20was%20taken%20active-active%20across%20two%20AWS%20regions%20with%20idempotency%20keys%2C%20conflict-free%20replication%2C%20and%20a%20tested%20RTO%20and%20RPO%2C%20so%20a%20full%20regional%20outage%20never%20double-charges%20a%20customer%20or%20loses%20a%20committed%20payment.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Multi-Region%20Active-Active%20for%20a%20Payments%20API%20https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fmulti-region-active-active-payments "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fmulti-region-active-active-payments&text=Multi-Region%20Active-Active%20for%20a%20Payments%20API "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fmulti-region-active-active-payments&title=Multi-Region%20Active-Active%20for%20a%20Payments%20API "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fmulti-region-active-active-payments&t=Multi-Region%20Active-Active%20for%20a%20Payments%20API "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fcase-studies%2Fpost%2Fmulti-region-active-active-payments&media=&description=How%20a%20money-movement%20API%20was%20taken%20active-active%20across%20two%20AWS%20regions%20with%20idempotency%20keys%2C%20conflict-free%20replication%2C%20and%20a%20tested%20RTO%20and%20RPO%2C%20so%20a%20full%20regional%20outage%20never%20double-charges%20a%20customer%20or%20loses%20a%20committed%20payment. "Share on Pinterest")[Email]() ## Comments ## You might also enjoy More posts on similar topics [![Migrating a Monolith to Kubernetes Without a Big-Bang Cutover](/_astro/hero.CAKh7bXG_Z1PnEBt.webp)](/case-studies/post/monolith-to-kubernetes-strangler-migration) ## [Migrating a Monolith to Kubernetes Without a Big-Bang Cutover](/case-studies/post/monolith-to-kubernetes-strangler-migration) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Cloud Native](/case-studies/categories/cloud-native) - [Architecture](/case-studies/categories/architecture) Almost every failed "let's move off the monolith" project shares one detail: the plan was a big-bang cutover. Rewrite in parallel, pick a weekend, flip the switch, and pray. This is the opposite of th [#Kubernetes](/case-studies/tags/kubernetes)[#EKS](/case-studies/tags/eks)[#Migration](/case-studies/tags/migration)+4 tags [read more](/case-studies/post/monolith-to-kubernetes-strangler-migration) [![Cutting a SaaS AWS Bill 41% Without Slowing Delivery](/_astro/hero.DJTB593d_Z1iASgs.webp)](/case-studies/post/aws-cost-optimization-saas-case-study) ## [Cutting a SaaS AWS Bill 41% Without Slowing Delivery](/case-studies/post/aws-cost-optimization-saas-case-study) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud Computing](/case-studies/categories/cloud-computing) - [DevOps](/case-studies/categories/devops) - [Cloud Native](/case-studies/categories/cloud-native) A growing SaaS ran on EKS with a full GitOps pipeline, and it was over its AWS budget nearly every month. The reflex from leadership was the usual one: freeze features until the bill comes down. That [#AWS](/case-studies/tags/aws)[#EKS](/case-studies/tags/eks)[#Kubernetes](/case-studies/tags/kubernetes)+7 tags [read more](/case-studies/post/aws-cost-optimization-saas-case-study) [![Zero-Downtime PostgreSQL Major-Version Upgrade at Scale](/_astro/hero.C03RcOLI_141vsK.webp)](/case-studies/post/zero-downtime-postgres-upgrade) ## [Zero-Downtime PostgreSQL Major-Version Upgrade at Scale](/case-studies/post/zero-downtime-postgres-upgrade) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Databases](/case-studies/categories/databases) - [Cloud Computing](/case-studies/categories/cloud-computing) A multi-terabyte PostgreSQL 12 database was reaching end of life, and the business ran around the clock, so the usual answer of "schedule a maintenance window" was off the table. We upgraded it to Pos [#PostgreSQL](/case-studies/tags/postgresql)[#Logical Replication](/case-studies/tags/logical-replication)[#Zero Downtime](/case-studies/tags/zero-downtime)+3 tags [read more](/case-studies/post/zero-downtime-postgres-upgrade) [![QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog](/_astro/hero.BfjMKoMg_ZI2zvl.webp)](/case-studies/post/quenchworks-zero-cve-catalog) ## [QuenchWorks: Building a 0-CVE Container Image and Helm Chart Catalog](/case-studies/post/quenchworks-zero-cve-catalog) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Security](/case-studies/categories/security) - [DevOps](/case-studies/categories/devops) - [Cloud Native](/case-studies/categories/cloud-native) When Bitnami moved its long-trusted catalog behind a paid tier, thousands of teams woke up to a supply-chain problem they didn't choose. The free images they had pinned in production would stop gettin [#Containers](/case-studies/tags/containers)[#Wolfi](/case-studies/tags/wolfi)[#Helm](/case-studies/tags/helm)+5 tags [read more](/case-studies/post/quenchworks-zero-cve-catalog) [![Building an Internal Developer Platform on Backstage and GitOps](/_astro/hero.Dq3xrist_1gdSPN.webp)](/case-studies/post/internal-developer-platform-backstage-gitops) ## [Building an Internal Developer Platform on Backstage and GitOps](/case-studies/post/internal-developer-platform-backstage-gitops) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/case-studies/categories/devops) - [Platform Engineering](/case-studies/categories/platform-engineering) - [Cloud Native](/case-studies/categories/cloud-native) Product teams were spending more time waiting on the platform team than building features. Spinning up a new service meant opening a ticket and waiting for someone to provision a repo, wire up CI, wri [#Backstage](/case-studies/tags/backstage)[#GitOps](/case-studies/tags/gitops)[#Argo CD](/case-studies/tags/argo-cd)+4 tags [read more](/case-studies/post/internal-developer-platform-backstage-gitops) 5 related posts