A practice prompt we wrote. No company or candidate report names it, so it carries no company tag.
How to answer
There are two jobs here, and the answer that loses the point does only the first: stop new duplicates, and repair the ones already written.
- Reframe the cause. Treat webhooks as delivered at least once. Stripe’s webhook documentation says endpoints might occasionally receive the same event more than once, and that it sometimes sends two separate events for the same object, and tells you to guard against both. So the bug is in the receiver, not the sender. Ask what “processed” did: credited a balance, shipped goods, sent an email?
- Find which duplicate it was. A redelivery of the same event (our handler timed out or returned an error after doing the work), a race between concurrent deliveries, or two different events describing the same payment. The first two share an event id; the third doesn’t, so it decides which key you dedupe on.
- Make the receiver idempotent with the database, not a check. Record the event id under a unique constraint in the same transaction as the effect, and put a second unique constraint on the business key the effect writes (one credit per payment). Act on one canonical event type per business fact. Treat the event as a notification: fetch the payment’s current state from the provider’s API and apply the transition to paid once, which also survives events arriving out of order. Acknowledge fast; move slow work to a queue.
- Keep side effects single. Emails and calls to other systems go through an outbox table with the same key, and the key is passed to any API that accepts one.
- Reconcile. Query for payments credited more than once, compare against the provider’s own list of payments, reverse extras with compensating entries rather than deletes, and tell the customer what you found and what you changed.
Say “at least once” early. It shows you know duplicates are normal, not an incident to blame on the provider.
Follow-ups
What the interviewer may ask next, once your first answer is on the table.
- The duplicate had a different event id from the original. Does your fix still catch it?
- Processing the event sends a receipt email and calls the customer’s ERP. How do those stay single?
- Your dedupe table grows forever. When can you delete rows from it?
Where answers go wrong
- Checking “have we seen this event id?” with a read and then processing. Two deliveries that arrive together both pass the read.
- Fixing only the handler, with nothing said about the duplicates already in the ledger or about telling the customer.
Answer this in two minutes
Write the answer you would say out loud. The clock starts with your first word.
Compare with the model answer
Model answer
The first minute and a half. “Webhooks are at-least-once, so the fix is in our receiver, not a complaint to the provider. If duplicated money can still be paid out, I pause payouts on the affected accounts first. Then I’d find which duplicate it was: a redelivery, a race, or two events for one payment. Then I’d make the database refuse the second one, with a unique event id in the same transaction as the credit and one live credit per payment. Side effects go through an outbox. Then I’d reverse the existing duplicates with compensating entries and tell the customer before they find it.”
If they want the detail:
“Before any design, I’d ask whose webhook it is: ours receiving from the payment provider, or ours sending to the customer? I’ll assume we receive. And is money still leaving? If a duplicated credit can be withdrawn or paid out, I’d pause payouts on the affected accounts and tell their finance lead today. Then I fix the cause.”
“First I’d pull both deliveries from our logs and from the provider’s delivery log, which shows each attempt and the status code we returned. A timeout or an HTTP 500 from us on the first attempt explains a redelivery. Same event id means a redelivery or a race. Different ids for the same payment means two separate events, which Stripe’s documentation says can happen even for the same event type, or that we act on two event types, such as charge.succeeded and payment_intent.succeeded. So I pick one canonical event for ‘money arrived’, ignore the other, and key the ledger on the payment, not the event.”
“The schema carries the guarantees. Event ids are unique, and a payment gets at most one live credit:”
CREATE TABLE webhook_events (
event_id text PRIMARY KEY,
type text NOT NULL,
payload jsonb NOT NULL,
received_at timestamptz NOT NULL DEFAULT now()
);
ALTER TABLE ledger_entries
ADD COLUMN superseded_by bigint REFERENCES ledger_entries (id);
CREATE UNIQUE INDEX CONCURRENTLY one_credit_per_payment
ON ledger_entries (payment_id)
WHERE kind = 'credit' AND superseded_by IS NULL;
CREATE TABLE outbox (
id bigserial PRIMARY KEY,
topic text NOT NULL,
dedupe_key text NOT NULL,
body jsonb NOT NULL,
sent_at timestamptz,
UNIQUE (topic, dedupe_key)
);“The handler verifies the signature, then does everything in one transaction:”
def handle(raw_body: bytes, signature: str) -> int:
event = verify(raw_body, signature) # bad signature: return 400, write nothing
with db.transaction() as tx:
fresh = tx.execute(
"INSERT INTO webhook_events (event_id, type, payload) "
"VALUES (%s, %s, %s) ON CONFLICT (event_id) DO NOTHING",
(event.id, event.type, json.dumps(event.data)),
).rowcount
if not fresh:
return 200 # already handled; acknowledge so the provider stops
if event.type == "payment_intent.succeeded": # the one canonical event
pi = event.data["object"]
credited = tx.execute(
"INSERT INTO ledger_entries (account_id, payment_id, kind, amount_cents) "
"VALUES (%s, %s, 'credit', %s) "
"ON CONFLICT (payment_id) WHERE kind = 'credit' AND superseded_by IS NULL "
"DO NOTHING",
(account_for(pi), pi["id"], pi["amount_received"]),
).rowcount
if credited:
tx.execute(
"INSERT INTO outbox (topic, dedupe_key, body) VALUES "
"('receipt_email', %s, %s) ON CONFLICT (topic, dedupe_key) DO NOTHING",
(pi["id"], json.dumps({"payment_id": pi["id"]})),
)
return 200“Why this is race-safe: if two deliveries arrive together, the second insert of the same event id waits on the first transaction’s uncommitted row, then does nothing once it commits. No read-then-write gap. Why the second constraint: a different event for the same payment gets past the first insert and stops at the ledger.”
“I trust this event’s payload because succeeded is a terminal status for a PaymentIntent, so a late delivery can’t move the payment backwards. Stripe’s documentation says it doesn’t guarantee delivery order, so for events where order matters, such as refunds and disputes, the queue worker fetches the object’s current state from the API before acting.”
“The ledger’s ON CONFLICT needs the index to exist, and the index won’t build while old duplicates exist. So it ships in four steps. One: the handler with the event-id insert only, which stops redeliveries and races right away. Two: reconciliation. Three: CREATE UNIQUE INDEX CONCURRENTLY, outside a transaction, and I check pg_index.indisvalid afterwards, because a duplicate that slips in during the build leaves an invalid index that I drop and rebuild. Four: the handler version with the ledger ON CONFLICT.”
“The handler only writes rows, so it returns quickly. A slow handler times out, and the provider treats a timeout like an error: a failed delivery it sends again.”
“The email leaves through the outbox, and the worker sends the payment id as the to anything downstream that accepts one. The worker is still at-least-once: if it crashes after sending and before marking the row done, the email goes twice, and for a receipt I accept that. For an ERP with no idempotency key, I write the payment id into a reference field on the ERP record and look it up before creating, and the nightly reconciliation compares ERP records against our ledger.”
“Reconciliation:”
SELECT payment_id,
array_agg(id ORDER BY id) AS entry_ids,
sum(amount_cents) AS credited_cents
FROM ledger_entries
WHERE kind = 'credit' AND superseded_by IS NULL
GROUP BY payment_id
HAVING count(*) > 1;“I keep the earliest credit per payment. For each later one I insert a reversal entry and set the duplicate’s superseded_by to the reversal’s id, so nothing is deleted and the audit trail stays intact. Each payment’s fix is one transaction, and the filter makes the script safe to rerun, because a payment that has been fixed drops out of the result. superseded_by is the one column that changes after posting, and it exists only so the partial index can exclude reversed credits. If their ledger must be strictly append-only, I’d keep reversals in their own table and enforce one credit per payment in the handler’s transaction instead, by locking the payment’s row with SELECT ... FOR UPDATE before checking for a live credit. Then I compare our credits against the provider’s list API for the same window, which also catches the other direction: payments we never credited.”
“Last, I’d send the customer the list of affected accounts, the amounts reversed and the change we made, before they find it in their own books. If any of that money had already been paid out, that’s a conversation with them about recovery, not a query I run.”
“The dedupe table can be pruned once rows are older than the longest time the provider can still send an event, manual resends included, with margin. For Stripe, its webhook documentation says live-mode automatic retries run for up to 3 days, a Dashboard resend works for 15 days after the event was created, and a CLI resend for 30 days, so I’d keep rows for 45 days. The ledger constraint keeps protecting the money after that.”