How to build webhook consumers that survive retries, duplicates, and outages

Verify, then persist, then acknowledge

Signature verification happens on the raw request body, before any parsing or normalisation, using a constant-time comparison. Any framework that reads and re-serialises the body before you verify will break the signature and tempt someone to skip verification.

Once verified, write the raw payload to storage with a unique constraint on the provider's event id, then return 2xx. Providers time out generously but not indefinitely, and slow synchronous processing is the most common cause of duplicate deliveries in the first place.

Idempotency in practice

• Key on the provider event id where one exists; otherwise hash the stable fields of the payload.
• Let the database enforce uniqueness. Application-level 'have I seen this?' checks race under concurrency.
• Make downstream effects idempotent too: upserts rather than inserts, and external calls carrying their own idempotency keys.

Ordering is not guaranteed

Assume events arrive out of order. Carry a version or timestamp from the provider and discard state transitions that move backwards. Where ordering genuinely matters, serialise per entity — one queue partition per record key — rather than globally, which destroys throughput.

Failure handling that a human can use

Retries need bounded attempts with exponential back-off and jitter, then a dead-letter store that keeps the full payload and the failure reason. A dead-letter queue nobody can inspect or replay from the UI is a silent data-loss mechanism.

Instrument three numbers: events received, events processed, and backlog age. Backlog age is the one that predicts incidents.