Kolo Bank Server

Problem

A bank backend has to stay correct while many requests hit the same account at once, clients retry, and workers die mid-payment. Unit tests that pass one request at a time don't show whether that holds. I needed measured evidence for each protection, and to find out where it didn't hold.

What I built

• Balances are never stored: they are summed from an append-only ledger, and Postgres triggers reject unbalanced transactions and lock the account row before every debit.
• Idempotency keys are written in the same transaction as the money movement, and a hold commits together with the row that owns it, under a per-key advisory lock.
• External payments reserve money with a hold, then capture or release it. Workers claim rows with FOR UPDATE SKIP LOCKED, and a resolver retries stuck transfers under the same keys.
• Every external dependency (payment rails, KYC, card network) is a deterministic in-process stub, so failures and timeouts can be reproduced in tests.
• A stress test suite that measures each protection on and off, using test-only switches (a swapped SQL function, a build tag), plus a worker binary that is SIGKILLed mid-batch.
• Signed webhooks with exponential backoff, delivered at least once by concurrent workers.

Results

• 500 concurrent transfers against an account funded for 10: 0 overdrafts, against 9 to 46 per run with row locking turned off.
• 8,000 mixed concurrent operations: 0 unbalanced transactions, and every account balance matched an independent model.
• 100 concurrent copies of one request with the same key: 1 posting and 1 hold. Before the fix the same test left 99 orphaned holds and 99 HTTP 500s.
• Worker SIGKILLed 25 times during 400 transfers: every transfer reached a final state 111 to 117 s after the last restart, with 0 double postings.
• Found and fixed 5 bugs with the tests, including a deadlock that failed 3.1% to 4.6% of mixed operations.
• k6 on 4 shared vCPUs: 287–370 req/s before the fixes and 409–458 after, with 0 errors at up to 200 virtual users.

What I learned

• Idempotency only holds if the side effect and its owning record commit together; a check followed by a separate write leaked held money on every retry.
• Crash safety has to be tested at each step boundary, because random kills rarely land in the few milliseconds where the bugs were.
• Summing a balance from history on every request makes latency grow with account age: about 6 ms at 2 entries, about 51 ms at 20,000.