Staff infrastructure engineer in San Francisco. A decade on auth, payments, and the seams between them. Currently shaping the auth-service-v2 rewrite — opinions and receipts.
Overriding the consensus to put idempotency in the application service. Pushing it to Cloudflare Workers with a 24h key store. Betting the storage cost stays under $400/month at 10× current volume, and that we eliminate at least 80% of duplicate-charge incidents within a quarter.
If key-store consistency lags, we double-charge customers — and the failure mode is silent until support tickets pile up. Mitigation: shadow-mode for two weeks against the existing service; alert on key-store skew above 50ms.
Right call. Eliminated 94% of duplicate-charge incidents in Q4 — better than the bet. Storage cost peaked at $312/month under Black Friday load (2.3× higher than predicted; the bet would still have held). One thing I underestimated: the debugging story for “missing” keys turned out harder than expected because the key-store doesn’t surface in app logs. Spent ~15 hours on observability we should have built up front.