Why webhooks fail in production

Most lost webhooks aren’t bugs in the sender. They are ordinary operational events your endpoint wasn’t built to survive.

1. Your endpoint was down

Deploys, restarts and cold starts all produce a window where the sender gets a 5xx or a timeout. Some providers retry for hours, others give up after a few attempts. A gateway accepts the event immediately and holds it until you are back.

2. The same event arrives twice

Senders retry when they don’t see a fast 2xx, so duplicates are normal. Handle them with an idempotency key (usually the event ID), or let the gateway drop repeats before they reach you.

3. A burst exceeds what you can process

A flash sale can send thousands of events per second. Queueing and rate limiting turn the spike into a steady stream your service can digest.

4. Events arrive out of order

order.updated can land before order.created. Either make handlers order-independent or deliver per-key in sequence.

5. Nobody can see what happened

Without a timeline for each event, “did we get it?” turns into log archaeology. Keep a record of every attempt, response and retry.

Next: read the delivery policy docs to see how retries and circuit breakers are configured.